SU\G(𝔸)/K·U
Sign Up for free Log In

Author Description

Login to generate an author description

Ask a Question About This Mathematician

Gemini: A Family of Highly Capable Multimodal Models

2023-01-01
Gemini Team , Rohan Anil , Sebastian Borgeaud +7 more
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano … This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models responsibly to users through services including Gemini, Gemini Advanced, Google AI Studio, and Cloud Vertex AI.
View PDF Chat

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

2024-03-08
Gemini Team , Machel Reid , Nikolay Savinov +7 more
In this report, we present the latest model of the Gemini family, Gemini 1.5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from … In this report, we present the latest model of the Gemini family, Gemini 1.5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. Gemini 1.5 Pro achieves near-perfect recall on long-context retrieval tasks across modalities, improves the state-of-the-art in long-document QA, long-video QA and long-context ASR, and matches or surpasses Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 2.1 (200k) and GPT-4 Turbo (128k). Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
View PDF Chat

Description-Driven Task-Oriented Dialog Modeling

2022-01-01
Jeffrey Zhao , Raghav Gupta , Yuan Cao +6 more
Task-oriented dialogue (TOD) systems are required to identify key information from conversations for the completion of given tasks. Such information is conventionally specified in terms of intents and slots contained … Task-oriented dialogue (TOD) systems are required to identify key information from conversations for the completion of given tasks. Such information is conventionally specified in terms of intents and slots contained in task-specific ontology or schemata. Since these schemata are designed by system developers, the naming convention for slots and intents is not uniform across tasks, and may not convey their semantics effectively. This can lead to models memorizing arbitrary patterns in data, resulting in suboptimal performance and generalization. In this paper, we propose that schemata should be modified by replacing names or notations entirely with natural language descriptions. We show that a language description-driven system exhibits better understanding of task specifications, higher performance on state tracking, improved data efficiency, and effective zero-shot transfer to unseen tasks. Following this paradigm, we present a simple yet effective Description-Driven Dialog State Tracking (D3ST) model, which relies purely on schema descriptions and an "index-picking" mechanism. We demonstrate the superiority in quality, data efficiency and robustness of our approach as measured on the MultiWOZ (Budzianowski et al.,2018), SGD (Rastogi et al., 2020), and the recent SGD-X (Lee et al., 2021) benchmarks.
View PDF Chat

Learning to Infer Entities, Properties and their Relations from Clinical Conversations

2019-01-01
Nan Du , Mingqiu Wang , Linh Tran +2 more
Nan Du, Mingqiu Wang, Linh Tran, Gang Lee, Izhak Shafran. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural … Nan Du, Mingqiu Wang, Linh Tran, Gang Lee, Izhak Shafran. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
View PDF Chat

Adaptive Lasso estimators for ultrahigh dimensional generalized linear models

2014-03-03
Mingqiu Wang , Xiuli Wang

Bridge estimation for generalized linear models with a diverging number of parameters

2010-07-09
Mingqiu Wang , Lixin Song , Xiaoguang Wang

Variable selection for high-dimensional generalized linear models with the weighted elastic-net procedure

2015-09-01
Xiuli Wang , Mingqiu Wang
High-dimensional data arise frequently in modern applications such as biology, chemometrics, economics, neuroscience and other scientific fields. The common features of high-dimensional data are that many of predictors may not … High-dimensional data arise frequently in modern applications such as biology, chemometrics, economics, neuroscience and other scientific fields. The common features of high-dimensional data are that many of predictors may not be significant, and there exists high correlation among predictors. Generalized linear models, as the generalization of linear models, also suffer from the collinearity problem. In this paper, combining the nonconvex penalty and ridge regression, we propose the weighted elastic-net to deal with the variable selection of generalized linear models on high dimension and give the theoretical properties of the proposed method with a diverging number of parameters. The finite sample behavior of the proposed method is illustrated with simulation studies and a real data example.

Adaptive group Lasso for high-dimensional generalized linear models

2017-02-03
Mingqiu Wang , Guo‐Liang Tian

A note on the one-step estimator for ultrahigh dimensionality

2013-10-04
Mingqiu Wang , Xiuli Wang , Xiaoguang Wang

SCAD-Penalized Least Absolute Deviation Regression in High-Dimensional Models

2014-10-30
Mingqiu Wang , Lixin Song , Guo‐Liang Tian
When outliers and/or heavy-tailed errors exist in linear models, the least absolute deviation (LAD) regression is a robust alternative to the ordinary least squares regression. Existing variable-selection methods in linear … When outliers and/or heavy-tailed errors exist in linear models, the least absolute deviation (LAD) regression is a robust alternative to the ordinary least squares regression. Existing variable-selection methods in linear models based on LAD regression either only consider the finite number of predictors or lack the oracle property associated with the estimator. In this article, we focus on the variable selection via LAD regression with a diverging number of parameters. The rate of convergence of the LAD estimator with the smoothly clipped absolute deviation (SCAD) penalty function is established. Furthermore, we demonstrate that, under certain regularity conditions, the penalized estimator with a properly selected tuning parameter enjoys the oracle property. In addition, the rank correlation screening method originally proposed by Li et al. (2011 Li, G.R., Peng, H., Zhu, L.X. (2011). Nonconcave penalized M-estimation with a diverging number of parameters. Statistica Sinica 21:391–419.[Web of Science ®] , [Google Scholar]) is applied to deal with ultrahigh dimensional data. Simulation studies are conducted for revealing the finite sample performance of the estimator. We further illustrate the proposed methodology by a real example.

Variable selection in the high-dimensional continuous generalized linear model with current status data

2013-09-20
Guo‐Liang Tian , Mingqiu Wang , Lixin Song
In survival studies, current status data are frequently encountered when some individuals in a study are not successively observed. This paper considers the problem of simultaneous variable selection and parameter … In survival studies, current status data are frequently encountered when some individuals in a study are not successively observed. This paper considers the problem of simultaneous variable selection and parameter estimation in the high-dimensional continuous generalized linear model with current status data. We apply the penalized likelihood procedure with the smoothly clipped absolute deviation penalty to select significant variables and estimate the corresponding regression coefficients. With a proper choice of tuning parameters, the resulting estimator is shown to be a root n/pn-consistent estimator under some mild conditions. In addition, we show that the resulting estimator has the same asymptotic distribution as the estimator obtained when the true model is known. The finite sample behavior of the proposed estimator is evaluated through simulation studies and a real example.
View PDF Chat

Adaptive group bridge estimation for high-dimensional partially linear models

2017-06-30
Xiuli Wang , Mingqiu Wang
This paper studies group selection for the partially linear model with a diverging number of parameters. We propose an adaptive group bridge method and study the consistency, convergence rate and … This paper studies group selection for the partially linear model with a diverging number of parameters. We propose an adaptive group bridge method and study the consistency, convergence rate and asymptotic distribution of the global adaptive group bridge estimator under regularity conditions. Simulation studies and a real example show the finite sample performance of our method.
View PDF Chat

Quadratic Approximation via the SCAD Penalty with a Diverging Number of Parameters

2014-06-09
Mingqiu Wang , Lixin Song , Xiaoguang Wang
The high-dimensional data arises in diverse fields of sciences, engineering and humanities. Variable selection plays an important role in dealing with high dimensional statistical modelling. In this article, we study … The high-dimensional data arises in diverse fields of sciences, engineering and humanities. Variable selection plays an important role in dealing with high dimensional statistical modelling. In this article, we study the variable selection of quadratic approximation via the smoothly clipped absolute deviation (SCAD) penalty with a diverging number of parameters. We provide a unified method to select variables and estimate parameters for various of high dimensional models. Under appropriate conditions and with a proper regularization parameter, we show that the estimator has consistency and sparsity, and the estimators of nonzero coefficients enjoy the asymptotic normality as they would have if the zero coefficients were known in advance. In addition, under some mild conditions, we can obtain the global solution of the penalized objective function with the SCAD penalty. Numerical studies and a real data analysis are carried out to confirm the performance of the proposed method.

Robust group non-convex estimations for high-dimensional partially linear models

2015-12-09
Mingqiu Wang , Guo‐Liang Tian
High-dimensional data with a group structure of variables arise always in many contemporary statistical modelling problems. Heavy-tailed errors or outliers in the response often exist in these data. We consider … High-dimensional data with a group structure of variables arise always in many contemporary statistical modelling problems. Heavy-tailed errors or outliers in the response often exist in these data. We consider robust group selection for partially linear models when the number of covariates can be larger than the sample size. The non-convex penalty function is applied to achieve both goals of variable selection and estimation in the linear part simultaneously, and we use polynomial splines to estimate the nonparametric component. Under regular conditions, we show that the robust estimator enjoys the oracle property. Simulation studies demonstrate the performance of the proposed method with samples of moderate size. The analysis of a real example illustrates that our method works well.

Restricted profile estimation for partially linear models with large-dimensional covariates

2017-04-28
Xiuli Wang , Shengli Zhao , Mingqiu Wang

Identification for semiparametric varying coefficient partially linear models

2013-02-18
Mingqiu Wang , Lixin Song

Nonconvex penalized ridge estimations for partially linear additive models in ultrahigh dimension

2015-03-11
Mingqiu Wang

Learning to Infer Entities, Properties and their Relations from Clinical Conversations

2019-01-01
Nan Du , Mingqiu Wang , Linh Tran +2 more
Recently we proposed the Span Attribute Tagging (SAT) Model (Du et al., 2019) to infer clinical entities (e.g., symptoms) and their properties (e.g., duration). It tackles the challenge of large … Recently we proposed the Span Attribute Tagging (SAT) Model (Du et al., 2019) to infer clinical entities (e.g., symptoms) and their properties (e.g., duration). It tackles the challenge of large label space and limited training data using a hierarchical two-stage approach that identifies the span of interest in a tagging step and assigns labels to the span in a classification step. We extend the SAT model to jointly infer not only entities and their properties but also relations between them. Most relation extraction models restrict inferring relations between tokens within a few neighboring sentences, mainly to avoid high computational complexity. In contrast, our proposed Relation-SAT (R-SAT) model is computationally efficient and can infer relations over the entire conversation, spanning an average duration of 10 minutes. We evaluate our model on a corpus of clinical conversations. When the entities are given, the R-SAT outperforms baselines in identifying relations between symptoms and their properties by about 32% (0.82 vs 0.62 F-score) and by about 50% (0.60 vs 0.41 F-score) on medications and their properties. On the more difficult task of jointly inferring entities and relations, the R-SAT model achieves a performance of 0.34 and 0.45 for symptoms and medications respectively, which is significantly better than 0.18 and 0.35 for the baseline model. The contributions of different components of the model are quantified using ablation analysis.

A Cholesky-based estimation for large-dimensional covariance matrices

2019-09-10
Xiaoning Kang , Chaoping Xie , Mingqiu Wang
This paper develops a new method to estimate a large-dimensional covariance matrix when the variables have no natural ordering among themselves. The modified Cholesky decomposition technique is used to provide … This paper develops a new method to estimate a large-dimensional covariance matrix when the variables have no natural ordering among themselves. The modified Cholesky decomposition technique is used to provide a set of estimates of the covariance matrix under multiple orderings of variables. The proposed estimator is in the form of a linear combination of these available estimates and the identity matrix. It is positive definite and applicable in large dimensions. The merits of the proposed estimator are demonstrated through the numerical study and a real data example by comparison with several existing methods.

The Medical Scribe: Corpus Development and Model Performance Analyses

2020-01-01
Izhak Shafran , Nan Du , Linh Tran +7 more
There is a growing interest in creating tools to assist in clinical note generation using the audio of provider-patient encounters. Motivated by this goal and with the help of providers … There is a growing interest in creating tools to assist in clinical note generation using the audio of provider-patient encounters. Motivated by this goal and with the help of providers and medical scribes, we developed an annotation scheme to extract relevant clinical concepts. We used this annotation scheme to label a corpus of about 6k clinical encounters. This was used to train a state-of-the-art tagging model. We report ontologies, labeling results, model performances, and detailed analyses of the results. Our results show that the entities related to medications can be extracted with a relatively high accuracy of 0.90 F-score, followed by symptoms at 0.72 F-score, and conditions at 0.57 F-score. In our task, we not only identify where the symptoms are mentioned but also map them to canonical forms as they appear in the clinical notes. Of the different types of errors, in about 19-38% of the cases, we find that the model output was correct, and about 17-32% of the errors do not impact the clinical note. Taken together, the models developed in this work are more useful than the F-scores reflect, making it a promising approach for practical applications.

Ensemble sparse estimation of covariance structure for exploring genetic disease data

2021-03-18
Xiaoning Kang , Mingqiu Wang

Bridge Estimators in the Partially Linear Model with High Dimensionality

2011-11-04
Mingqiu Wang , Lixin Song , Xiaoguang Wang
This article studies variable selection and parameter estimation in the partially linear model when the number of covariates in the linear part increases to infinity. Using the bridge penalty method, … This article studies variable selection and parameter estimation in the partially linear model when the number of covariates in the linear part increases to infinity. Using the bridge penalty method, we succeed in selecting the important covariates of the linear part. Under regularity conditions, we have shown that the bridge penalized estimator of the parametric part enjoys the oracle property. We also obtain the convergence rate of the estimator of the nonparametric part. Simulation studies show that the bridge estimator performs as well as the oracle estimator for the partially linear model. An application is analyzed to illustrate the bridge procedure.

Optimal subsampling for least absolute relative error estimators with massive data

2022-07-08
Min Ren , Shengli Zhao , Mingqiu Wang

SLM: Bridge the Thin Gap Between Speech and Text Foundation Models

2023-12-16
Mingqiu Wang , Wei Han , Izhak Shafran +7 more
We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM freezes the pretrained foundation … We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM freezes the pretrained foundation models to maximally preserves their capabilities, and only trains a simple adapter with just 1% (156M) of the foundation models' parameters. This adaptation not only leads SLM to achieve strong performance on conventional tasks such as automatic speech recognition (ASR) and automatic speech translation (AST), but also unlocks the novel capability of zero-shot instruction-following for more diverse tasks. Given a speech input and a text instruction, SLM is able to perform unseen generation tasks including contextual biasing ASR using real-time context, dialog generation, speech continuation, and question answering. Our approach demonstrates that the representational gap between pretrained speech and language models is narrower than one would expect, and can be bridged by a simple adaptation mechanism. As a result, SLM is not only efficient to train, but also inherits strong capabilities already present in foundation models of different modalities.
View PDF Chat

Combined-penalized likelihood estimations with a diverging number of parameters

2013-12-11
Ying Dong , Lixin Song , Mingqiu Wang +1 more
In the economics and biological gene expression study area where a large number of variables will be involved, even when the predictors are independent, as long as the dimension is … In the economics and biological gene expression study area where a large number of variables will be involved, even when the predictors are independent, as long as the dimension is high, the maximum sample correlation can be large. Variable selection is a fundamental method to deal with such models. The ridge regression performs well when the predictors are highly correlated and some nonconcave penalized thresholding estimators enjoy the nice oracle property. In order to provide a satisfactory solution to the collinearity problem, in this paper we report the combined-penalization (CP) mixed by the nonconcave penalty and ridge, with a diverging number of parameters. It is observed that the CP estimator with a diverging number of parameters can correctly select covariates with nonzero coefficients and can estimate parameters simultaneously in the presence of multicollinearity. Simulation studies and a real data example demonstrate the well performance of the proposed method.

Unsupervised Slot Schema Induction for Task-oriented Dialog

2022-01-01
Dian Yu , Mingqiu Wang , Yuan Cao +3 more
Dian Yu, Mingqiu Wang, Yuan Cao, Izhak Shafran, Laurent Shafey, Hagen Soltau. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language … Dian Yu, Mingqiu Wang, Yuan Cao, Izhak Shafran, Laurent Shafey, Hagen Soltau. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.
View PDF Chat

Structure identification for varying coefficient models with measurement errors based on kernel smoothing

2018-05-14
Mingqiu Wang , Peixin Zhao , Xiaoning Kang

Understanding Medical Conversations: Rich Transcription, Confidence Scores & Information Extraction.

2021-04-06
Hagen Soltau , Mingqiu Wang , Izhak Shafran +1 more
In this paper, we describe novel components for extracting clinically relevant information from medical conversations which will be available as Google APIs. We describe a transformer-based Recurrent Neural Network Transducer … In this paper, we describe novel components for extracting clinically relevant information from medical conversations which will be available as Google APIs. We describe a transformer-based Recurrent Neural Network Transducer (RNN-T) model tailored for long-form audio, which can produce rich transcriptions including speaker segmentation, speaker role labeling, punctuation and capitalization. On a representative test set, we compare performance of RNN-T models with different encoders, units and streaming constraints. Our transformer-based streaming model performs at about 20% WER on the ASR task, 6% WDER on the diarization task, 43% SER on periods, 52% SER on commas, 43% SER on question marks and 30% SER on capitalization. Our recognizer is paired with a confidence model that utilizes both acoustic and lexical features from the recognizer. The model performs at about 0.37 NCE. Finally, we describe a RNN-T based tagging model. The performance of the model depends on the ontologies, with F-scores of 0.90 for medications, 0.76 for symptoms, 0.75 for conditions, 0.76 for diagnosis, and 0.61 for treatments. While there is still room for improvement, our results suggest that these models are sufficiently accurate for practical applications.

Word-Level Confidence Estimation for RNN Transducers

2021-12-13
Mingqiu Wang , Hagen Soltau , Laurent El Shafey +1 more
Confidence estimate is an often requested feature in applications such as medical transcription where errors can impact patient care and the confidence estimate could be used to alert medical professionals … Confidence estimate is an often requested feature in applications such as medical transcription where errors can impact patient care and the confidence estimate could be used to alert medical professionals to verify potential errors in recognition. In this paper, we present a lightweight neural confidence model tailored for Automatic Speech Recognition (ASR) system with Recurrent Neural Network Transducers (RNN-T). Compared to other existing approaches, our model utilizes: (a) the time information associated with recognized words, which reduces the computational complexity, and (b) a simple and elegant trick for mapping between sub-word and word sequences. The mapping addresses the non-unique tokenization and token deletion problems while amplifying differences between confusable words. Through extensive empirical evaluations on two different long-form test sets, we demonstrate that the model achieves a performance of 0.4 Normalized Cross Entropy (NCE) and 0.05 Expected Calibration Error (ECE). It is robust across different ASR configurations, including target types (graphemes vs. morphemes), traffic conditions (streaming vs. non-streaming), and encoder types. We further discuss the importance of evaluation metrics to reflect practical applications and highlight the need for further work in improving Area Under the Curve (AUC) for Negative Precision Rate (NPV) and True Negative Rate (TNR).
View PDF Chat

Speech Aware Dialog System Technology Challenge (DSTC11)

2023-08-14
Hagen Soltau , Izhak Shafran , Mingqiu Wang +6 more
View PDF Chat

A Cholesky-based sparse covariance estimation with an application to genes data

2021-05-29
Chunshi Li , Mo Yang , Mingqiu Wang +2 more
The modified Cholesky decomposition (MCD) is a powerful tool for estimating a covariance matrix. The regularization can be conveniently imposed on the linear regressions to encourage the sparsity in the … The modified Cholesky decomposition (MCD) is a powerful tool for estimating a covariance matrix. The regularization can be conveniently imposed on the linear regressions to encourage the sparsity in the estimated covariance matrix to accommodate the high-dimensional data. In this paper, we propose a Cholesky-based sparse ensemble estimate for covariance matrix by averaging a set of Cholesky factor estimates obtained from multiple variable orderings used in the MCD. The sparse estimation is enabled by encouraging the sparsity in the Cholesky factor. The theoretical consistent property is established under some regular conditions. The merits of the proposed method are illustrated through simulation and a maize genes data set.

RNN Transducers for Nested Named Entity Recognition with constraints on alignment for long sequences

2022-01-01
Hagen Soltau , Izhak Shafran , Mingqiu Wang +1 more
Popular solutions to Named Entity Recognition (NER) include conditional random fields, sequence-to-sequence models, or utilizing the question-answering framework. However, they are not suitable for nested and overlapping spans with large … Popular solutions to Named Entity Recognition (NER) include conditional random fields, sequence-to-sequence models, or utilizing the question-answering framework. However, they are not suitable for nested and overlapping spans with large ontologies and for predicting the position of the entities. To fill this gap, we introduce a new model for NER task -- an RNN transducer (RNN-T). These models are trained using paired input and output sequences without explicitly specifying the alignment between them, similar to other seq-to-seq models. RNN-T models learn the alignment using a loss function that sums over all alignments. In NER tasks, however, the alignment between words and target labels are available from the human annotations. We propose a fixed alignment RNN-T model that utilizes the given alignment, while preserving the benefits of RNN-Ts such as modeling output dependencies. As a more general case, we also propose a constrained alignment model where users can specify a relaxation of the given input alignment and the model will learn an alignment within the given constraints. In other words, we propose a family of seq-to-seq models which can leverage alignments between input and target sequences when available. Through empirical experiments on a challenging real-world medical NER task with multiple nested ontologies, we demonstrate that our fixed alignment model outperforms the standard RNN-T model, improving F1-score from 0.70 to 0.74.
View PDF Chat

Doubly robust weighted composite quantile regression based on SCAD‐<i>L</i><sub>2</sub>

2021-12-20
Zhimiao Cao , Xiaoning Kang , Mingqiu Wang
In this article, a robust variable selection procedure based on the weighted composite quantile regression (WCQR) is proposed. Compared with the composite quantile regression (CQR), WCQR is robust to heavy‐tailed … In this article, a robust variable selection procedure based on the weighted composite quantile regression (WCQR) is proposed. Compared with the composite quantile regression (CQR), WCQR is robust to heavy‐tailed errors and outliers in the explanatory variables. For the choice of the weights in the WCQR, we employ a weighting scheme based on the principal component method. To select variables with grouping effect, we consider WCQR with SCAD‐ L 2 penalization. Furthermore, under some suitable assumptions, the theoretical properties, including the consistency and oracle property of the estimator, are established with a diverging number of parameters. In addition, we study the numerical performance of the proposed method in the case of ultrahigh‐dimensional data. Simulation studies and real examples are provided to demonstrate the superiority of our method over the CQR method when there are outliers in the explanatory variables and/or the random error is from a heavy‐tailed distribution.

Estimation and inference for transfer learning with high-dimensional quantile regression

2022-01-01
Jiayu Huang , Mingqiu Wang , Yuanshan Wu
Transfer learning has become an essential technique to exploit information from the source domain to boost performance of the target task. Despite the prevalence in high-dimensional data, heterogeneity and heavy … Transfer learning has become an essential technique to exploit information from the source domain to boost performance of the target task. Despite the prevalence in high-dimensional data, heterogeneity and heavy tails are insufficiently accounted for by current transfer learning approaches and thus may undermine the resulting performance. We propose a transfer learning procedure in the framework of high-dimensional quantile regression models to accommodate heterogeneity and heavy tails in the source and target domains. We establish error bounds of transfer learning estimator based on delicately selected transferable source domains, showing that lower error bounds can be achieved for critical selection criterion and larger sample size of source tasks. We further propose valid confidence interval and hypothesis test procedures for individual component of high-dimensional quantile regression coefficients by advocating a double transfer learning estimator, which is one-step debiased estimator for the transfer learning estimator wherein the technique of transfer learning is designed again. By adopting data-splitting technique, we advocate a transferability detection approach that guarantees to circumvent negative transfer and identify transferable sources with high probability. Simulation results demonstrate that the proposed method exhibits some favorable and compelling performances and the practical utility is further illustrated by analyzing a real example.

Heteroscedasticity identification and variable selection via multiple quantile regression

2023-08-05
Mingqiu Wang , Xiaoning Kang , Jiajuan Liang +2 more
AbstractHigh-dimensional data often display heteroscedasticity. If the heteroscedasticity is neglected in the regression model, it will produce inefficient inference for the regression coefficients. Quantile regression is not only robust to … AbstractHigh-dimensional data often display heteroscedasticity. If the heteroscedasticity is neglected in the regression model, it will produce inefficient inference for the regression coefficients. Quantile regression is not only robust to outliers, but also accommodates heteroscedasticity. This paper aims to simultaneously carry out variable selection and heteroscedasticity identification for the linear location-scale model under a unified framework. We develop a regularized multiple quantile regression approach simultaneously identifying the heteroscedasticity, seeking common features of quantile coefficients and eliminating irrelevant variables. We also establish the theoretical properties of the proposed method under some regularity conditions. Simulation studies are conducted to evaluate the finite sample performance of the proposed method, showing that it is able to identify the covariates that affect the variability of the response. We further apply the proposed method to analyse the Wage data.Keywords: Heteroscedasticitypenalty functionquantile regressionvariable selectionMathematics Subject Classifications: 62J0762F12 AcknowledgmentsThe authors would like to thank the Editor and two referees for the constructive suggestions that lead to a significant improvement over the article.Disclosure statementNo potential conflict of interest was reported by the author(s).Additional informationFundingThe research was supported by the National Natural Science Foundation of China [grant numbers 12271294, 12071248, 12071483, and 72232001], Natural Science Foundation of Liaoning Province [grant number 2022-MS-179], Department of Education of Liaoning Province [grant number LIKMZ20221565], and United International College (UIC) Start-up Research Fund [grant number R72021106], Science Foundation of Ministry of Education of China [grant number 20YJC910007].

Robust optimal subsampling based on weighted asymmetric least squares

2023-09-19
Min Ren , Shengli Zhao , Mingqiu Wang +1 more

AnyTOD: A Programmable Task-Oriented Dialog System

2023-01-01
Jeffrey Zhao , Yuan Cao , Raghav Gupta +6 more
Jeffrey Zhao, Yuan Cao, Raghav Gupta, Harrison Lee, Abhinav Rastogi, Mingqiu Wang, Hagen Soltau, Izhak Shafran, Yonghui Wu. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. … Jeffrey Zhao, Yuan Cao, Raghav Gupta, Harrison Lee, Abhinav Rastogi, Mingqiu Wang, Hagen Soltau, Izhak Shafran, Yonghui Wu. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
View PDF Chat

Identification and Estimation for Generalized Varying Coefficient Partially Linear Models

2016-09-27
Mingqiu Wang , Xiuli Wang , Muhammad Amin
The generalized varying coe fficient partially linear model (GVCPLM) enjoys the fl exibility of the generalized varying coeffi cient model and the parsimony and interpretability of the generalized linear model. … The generalized varying coefficient partially linear model (GVCPLM) enjoys the flexibility of the generalized varying coefficient model and the parsimony and interpretability of the generalized linear model. Statistical inference of GVCPLM is restricted with a condition that the components of varying and constant coefficients are known in advance. Alternatively, the current study is focused on the structure's identification of varying and constant coefficient for GVCPLM and it is based on the spline basis approximation and the group SCAD. This is proved that the proposed method can consistently determine the structure of the GVCPLM under certain conditions, which means that it can accurately choose the varying and constant coefficients precisely. Simulation studies and a real data application are conducted to assess the infinite sample performance of the proposed method.
View PDF Chat

Profile Statistical Inference for Partially Linear Additive Models with a Diverging Number of Parameters

2019-12-01
Xiuli Wang , Shengli Zhao , Mingqiu Wang

M-estimation of the mixed-type generalized linear model

2018-07-30
Ying Dong , Lixin Song , Mingqiu Wang +1 more
To investigate the features of the individual from the mixedtype model, a novel model, named the mixed-type generalized linear model, is proposed firstly in this work, which is verified to … To investigate the features of the individual from the mixedtype model, a novel model, named the mixed-type generalized linear model, is proposed firstly in this work, which is verified to be realistic and useful. We consider the robustness of M-estimation to estimate the unknown parameters of the mixed-type generalized linear model. By applying the law of large numbers and the central limit theorem, the consistency and asymptotic normality of the M-estimation for the mixed-type generalized linear model are proved with regularity assumptions. At last, in order to evaluate the finite sample performance of the estimator for the new model, several applied instances are presented, which show the good performance of the estimator.
View PDF Chat

Group selection via adjusted weighted least absolute deviation regression

2020-04-10
Xiuli Wang , Zhimiao Cao , Chao Liu +1 more

Statistical inference for semiparametric varying-coefficient partially linear models with a diverging number of components

2020-01-01
Mingqiu Wang , Guo‐Liang Tian , Yin Liu

A new variant of the parallel regression model with variable selection in surveys with sensitive attribute

2020-11-01
Yin Liu , Guo‐Liang Tian , Mingqiu Wang

Knowledge-grounded Dialog State Tracking

2022-01-01
Dian Yu , Mingqiu Wang , Yuan Cao +3 more
Knowledge (including structured knowledge such as schema and ontology and unstructured knowledge such as web corpus) is a critical part of dialog understanding, especially for unseen tasks and domains. Traditionally, … Knowledge (including structured knowledge such as schema and ontology and unstructured knowledge such as web corpus) is a critical part of dialog understanding, especially for unseen tasks and domains. Traditionally, such domain-specific knowledge is encoded implicitly into model parameters for the execution of downstream tasks, which makes training inefficient. In addition , such models are not easily transferable to new tasks with different schemas. In this work, we propose to perform dialog state tracking grounded on knowledge encoded externally. We query relevant knowledge of various forms based on the dialog context where such information can grounds the prediction of dialog states. We demonstrate superior performance of our proposed method over strong baselines, especially in the few-shot learning setting.
View PDF Chat

R2D2: Relational Text Decoding with Transformers

2021-01-01
Aryan Arbabi , Mingqiu Wang , Laurent El Shafey +2 more
We propose a novel framework for modeling the interaction between graphical structures and the natural language text associated with their nodes and edges. Existing approaches typically fall into two categories. … We propose a novel framework for modeling the interaction between graphical structures and the natural language text associated with their nodes and edges. Existing approaches typically fall into two categories. On group ignores the relational structure by converting them into linear sequences and then utilize the highly successful Seq2Seq models. The other side ignores the sequential nature of the text by representing them as fixed-dimensional vectors and apply graph neural networks. Both simplifications lead to information loss. Our proposed method utilizes both the graphical structure as well as the sequential nature of the texts. The input to our model is a set of text segments associated with the nodes and edges of the graph, which are then processed with a transformer encoder-decoder model, equipped with a self-attention mechanism that is aware of the graphical relations between the nodes containing the segments. This also allows us to use BERT-like models that are already trained on large amounts of text. While the proposed model has wide applications, we demonstrate its capabilities on data-to-text generation tasks. Our approach compares favorably against state-of-the-art methods in four tasks without tailoring the model architecture. We also provide an early demonstration in a novel practical application -- generating clinical notes from the medical entities mentioned during clinical visits.

Word-level confidence estimation for RNN transducers

2021-01-01
Mingqiu Wang , Hagen Soltau , Laurent El Shafey +1 more
Confidence estimate is an often requested feature in applications such as medical transcription where errors can impact patient care and the confidence estimate could be used to alert medical professionals … Confidence estimate is an often requested feature in applications such as medical transcription where errors can impact patient care and the confidence estimate could be used to alert medical professionals to verify potential errors in recognition. In this paper, we present a lightweight neural confidence model tailored for Automatic Speech Recognition (ASR) system with Recurrent Neural Network Transducers (RNN-T). Compared to other existing approaches, our model utilizes: (a) the time information associated with recognized words, which reduces the computational complexity, and (b) a simple and elegant trick for mapping between sub-word and word sequences. The mapping addresses the non-unique tokenization and token deletion problems while amplifying differences between confusable words. Through extensive empirical evaluations on two different long-form test sets, we demonstrate that the model achieves a performance of 0.4 Normalized Cross Entropy (NCE) and 0.05 Expected Calibration Error (ECE). It is robust across different ASR configurations, including target types (graphemes vs. morphemes), traffic conditions (streaming vs. non-streaming), and encoder types. We further discuss the importance of evaluation metrics to reflect practical applications and highlight the need for further work in improving Area Under the Curve (AUC) for Negative Precision Rate (NPV) and True Negative Rate (TNR).
View PDF Chat

Knowledge-grounded Dialog State Tracking

2022-01-01
Dian Yu , Mingqiu Wang , Yuan Cao +3 more
Knowledge (including structured knowledge such as schema and ontology, and unstructured knowledge such as web corpus) is a critical part of dialog understanding, especially for unseen tasks and domains. Traditionally, … Knowledge (including structured knowledge such as schema and ontology, and unstructured knowledge such as web corpus) is a critical part of dialog understanding, especially for unseen tasks and domains. Traditionally, such domain-specific knowledge is encoded implicitly into model parameters for the execution of downstream tasks, which makes training inefficient. In addition, such models are not easily transferable to new tasks with different schemas. In this work, we propose to perform dialog state tracking grounded on knowledge encoded externally. We query relevant knowledge of various forms based on the dialog context where such information can ground the prediction of dialog states. We demonstrate superior performance of our proposed method over strong baselines, especially in the few-shot learning setting.
View PDF Chat

Speech Aware Dialog System Technology Challenge (DSTC11)

2022-01-01
Hagen Soltau , Izhak Shafran , Mingqiu Wang +6 more
Most research on task oriented dialog modeling is based on written text input. However, users interact with practical dialog systems often using speech as input. Typically, systems convert speech into … Most research on task oriented dialog modeling is based on written text input. However, users interact with practical dialog systems often using speech as input. Typically, systems convert speech into text using an Automatic Speech Recognition (ASR) system, introducing errors. Furthermore, these systems do not address the differences in written and spoken language. The research on this topic is stymied by the lack of a public corpus. Motivated by these considerations, our goal in hosting the speech-aware dialog state tracking challenge was to create a public corpus or task which can be used to investigate the performance gap between the written and spoken forms of input, develop models that could alleviate this gap, and establish whether Text-to-Speech-based (TTS) systems is a reasonable surrogate to the more-labor intensive human data collection. We created three spoken versions of the popular written-domain MultiWoz task -- (a) TTS-Verbatim: written user inputs were converted into speech waveforms using a TTS system, (b) Human-Verbatim: humans spoke the user inputs verbatim, and (c) Human-paraphrased: humans paraphrased the user inputs. Additionally, we provided different forms of ASR output to encourage wider participation from teams that may not have access to state-of-the-art ASR systems. These included ASR transcripts, word time stamps, and latent representations of the audio (audio encoder outputs). In this paper, we describe the corpus, report results from participating teams, provide preliminary analyses of their results, and summarize the current state-of-the-art in this domain.

AnyTOD: A Programmable Task-Oriented Dialog System

2022-01-01
Jeffrey Zhao , Yuan Cao , Raghav Gupta +6 more
We propose AnyTOD, an end-to-end, zero-shot task-oriented dialog (TOD) system capable of handling unseen tasks without task-specific training. We view TOD as a program executed by a language model (LM), … We propose AnyTOD, an end-to-end, zero-shot task-oriented dialog (TOD) system capable of handling unseen tasks without task-specific training. We view TOD as a program executed by a language model (LM), where program logic and ontology is provided by a designer as a schema. To enable generalization to unseen schemas and programs without prior training, AnyTOD adopts a neuro-symbolic approach. A neural LM keeps track of events occurring during a conversation and a symbolic program implementing the dialog policy is executed to recommend next actions AnyTOD should take. This approach drastically reduces data annotation and model training requirements, addressing the enduring challenge of rapidly adapting a TOD system to unseen tasks and domains. We demonstrate state-of-the-art results on STAR, ABCD and SGD benchmarks. We also demonstrate strong zero-shot transfer ability in low-resource settings, such as zero-shot on MultiWOZ. In addition, we release STARv2, an updated version of the STAR dataset with richer annotations, for benchmarking zero-shot end-to-end TOD models.

Unsupervised Slot Schema Induction for Task-oriented Dialog

2022-01-01
Dian Yu , Mingqiu Wang , Yuan Cao +3 more
Carefully-designed schemas describing how to collect and annotate dialog corpora are a prerequisite towards building task-oriented dialog systems. In practical applications, manually designing schemas can be error-prone, laborious, iterative, and … Carefully-designed schemas describing how to collect and annotate dialog corpora are a prerequisite towards building task-oriented dialog systems. In practical applications, manually designing schemas can be error-prone, laborious, iterative, and slow, especially when the schema is complicated. To alleviate this expensive and time consuming process, we propose an unsupervised approach for slot schema induction from unlabeled dialog corpora. Leveraging in-domain language models and unsupervised parsing structures, our data-driven approach extracts candidate slots without constraints, followed by coarse-to-fine clustering to induce slot types. We compare our method against several strong supervised baselines, and show significant performance improvement in slot schema induction on MultiWoz and SGD datasets. We also demonstrate the effectiveness of induced schemas on downstream applications including dialog state tracking and response generation.
View PDF Chat

Sparse Robust Weighted Expectile Screening for Ultra-High-Dimensional Data

2025-04-28
Xianjun Wu , Peisong Han , Mingqiu Wang
This paper investigates robust feature screening for ultra-high dimensional data in the presence of outliers and heterogeneity. Considering the susceptibility of likelihood methods to outliers, we propose a Sparse Robust … This paper investigates robust feature screening for ultra-high dimensional data in the presence of outliers and heterogeneity. Considering the susceptibility of likelihood methods to outliers, we propose a Sparse Robust Weighted Expectile Regression (SRoWER) method that combines the L2E criterion with expectile regression. By utilizing the IHT algorithm, our method effectively incorporates correlations of covariates and enables joint feature screening. The proposed approach demonstrates robustness against heavy-tailed errors and outliers in data. Simulation studies and a real data analysis are provided to demonstrate the superior performance of the SRoWER method when dealing with outlier-contaminated explanatory variables and/or heavy-tailed error distributions.

Corrected optimal subsampling for a class of generalized linear measurement error models

2025-01-01
Ruiyuan Chang , Xiuli Wang , Mingqiu Wang

Knowledge Graph Reasoning with Self-supervised Reinforcement Learning

2024-05-22
Ying Ma , Owen Burns , Mingqiu Wang +6 more
Reinforcement learning (RL) is an effective method of finding reasoning pathways in incomplete knowledge graphs (KGs). To overcome the challenges of a large action space, a self-supervised pre-training method is … Reinforcement learning (RL) is an effective method of finding reasoning pathways in incomplete knowledge graphs (KGs). To overcome the challenges of a large action space, a self-supervised pre-training method is proposed to warm up the policy network before the RL training stage. To alleviate the distributional mismatch issue in general self-supervised RL (SSRL), in our supervised learning (SL) stage, the agent selects actions based on the policy network and learns from generated labels; this self-generation of labels is the intuition behind the name self-supervised. With this training framework, the information density of our SL objective is increased and the agent is prevented from getting stuck with the early rewarded paths. Our self-supervised RL (SSRL) method improves the performance of RL by pairing it with the wide coverage achieved by SL during pretraining, since the breadth of the SL objective makes it infeasible to train an agent with that alone. We show that our SSRL model meets or exceeds current state-of-the-art results on all Hits@k and mean reciprocal rank (MRR) metrics on four large benchmark KG datasets. This SSRL method can be used as a plug-in for any RL architecture for a KGR task. We adopt two RL architectures, i.e., MINERVA and MultiHopKG as our baseline RL models and experimentally show that our SSRL model consistently outperforms both baselines on all of these four KG reasoning tasks. Full code for the paper available at https://github.com/owenonline/Knowledge-Graph-Reasoning-with-Self-supervised-Reinforcement-Learning.
View PDF Chat

Retrieval Augmented End-to-End Spoken Dialog Models

2024-03-18
Mingqiu Wang , Izhak Shafran , Hagen Soltau +4 more
We recently developed a joint speech and language model (SLM [1]) which fuses a pretrained foundational speech model and a large language model (LLM), while preserving the in-context learning capability … We recently developed a joint speech and language model (SLM [1]) which fuses a pretrained foundational speech model and a large language model (LLM), while preserving the in-context learning capability intrinsic to the pretrained LLM. In this paper, we apply SLM to dialog applications where the dialog states are inferred directly from the audio signal.Task-oriented dialogs often contain domain-specific entities, i.e., restaurants, hotels, train stations, and city names, which are difficult to recognize, however, critical for the downstream applications. Inspired by the RAG (retrieval-augmented generation) models, we propose a retrieval augmented SLM (ReSLM) that overcomes this weakness. We first train a retriever to retrieve text entities given audio inputs. The retrieved entities are then added as text inputs to the underlying LLM to bias model predictions. We evaluated ReSLM on speech MultiWoz task (DSTC-11 Challenge), and found that the retrieval augmentation boosts model performance, achieving joint goal accuracy (38.6% vs 32.7%), slot error rate (20.6% vs 24.8%) and ASR word error rate (5.5% vs 6.7%). While demonstrated on dialog state tracking, our approach is broadly applicable to speech tasks requiring custom contextual information or domain-specific entities.
View PDF Chat

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

2024-03-08
Gemini Team , Machel Reid , Nikolay Savinov +7 more
In this report, we present the latest model of the Gemini family, Gemini 1.5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from … In this report, we present the latest model of the Gemini family, Gemini 1.5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. Gemini 1.5 Pro achieves near-perfect recall on long-context retrieval tasks across modalities, improves the state-of-the-art in long-document QA, long-video QA and long-context ASR, and matches or surpasses Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 2.1 (200k) and GPT-4 Turbo (128k). Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
View PDF Chat

Subsampling for Big Data Linear Models with Measurement Errors

2024-03-07
Jiangshan Ju , Mingqiu Wang , Shengli Zhao
Subsampling algorithms for various parametric regression models with massive data have been extensively investigated in recent years. However, all existing studies on subsampling heavily rely on clean massive data. In … Subsampling algorithms for various parametric regression models with massive data have been extensively investigated in recent years. However, all existing studies on subsampling heavily rely on clean massive data. In practical applications, the observed covariates may suffer from inaccuracies due to measurement errors. To address the challenge of large datasets with measurement errors, this study explores two subsampling algorithms based on the corrected likelihood approach: the optimal subsampling algorithm utilizing inverse probability weighting and the perturbation subsampling algorithm employing random weighting assuming a perfectly known distribution. Theoretical properties for both algorithms are provided. Numerical simulations and two real-world examples demonstrate the effectiveness of these proposed methods compared to other uncorrected algorithms.
View PDF Chat

Retrieval Augmented End-to-End Spoken Dialog Models

2024-02-02
Mingqiu Wang , Izhak Shafran , Hagen Soltau +4 more
We recently developed SLM, a joint speech and language model, which fuses a pretrained foundational speech model and a large language model (LLM), while preserving the in-context learning capability intrinsic … We recently developed SLM, a joint speech and language model, which fuses a pretrained foundational speech model and a large language model (LLM), while preserving the in-context learning capability intrinsic to the pretrained LLM. In this paper, we apply SLM to speech dialog applications where the dialog states are inferred directly from the audio signal. Task-oriented dialogs often contain domain-specific entities, i.e., restaurants, hotels, train stations, and city names, which are difficult to recognize, however, critical for the downstream applications. Inspired by the RAG (retrieval-augmented generation) paradigm, we propose a retrieval augmented SLM (ReSLM) that overcomes this weakness. We first train a speech retriever to retrieve text entities mentioned in the audio. The retrieved entities are then added as text inputs to the underlying SLM to bias model predictions. We evaluated ReSLM on speech MultiWoz task (DSTC-11 challenge), and found that this retrieval augmentation boosts model performance, achieving joint goal accuracy (38.6% vs 32.7%), slot error rate (20.6% vs 24.8%) and ASR word error rate (5.5% vs 6.7%). While demonstrated on dialog state tracking, our approach is broadly applicable to other speech tasks requiring contextual information or domain-specific entities, such as contextual ASR with biasing capability.
View PDF Chat

SLM: Bridge the Thin Gap Between Speech and Text Foundation Models

2023-12-16
Mingqiu Wang , Wei Han , Izhak Shafran +7 more
We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM freezes the pretrained foundation … We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM freezes the pretrained foundation models to maximally preserves their capabilities, and only trains a simple adapter with just 1% (156M) of the foundation models' parameters. This adaptation not only leads SLM to achieve strong performance on conventional tasks such as automatic speech recognition (ASR) and automatic speech translation (AST), but also unlocks the novel capability of zero-shot instruction-following for more diverse tasks. Given a speech input and a text instruction, SLM is able to perform unseen generation tasks including contextual biasing ASR using real-time context, dialog generation, speech continuation, and question answering. Our approach demonstrates that the representational gap between pretrained speech and language models is narrower than one would expect, and can be bridged by a simple adaptation mechanism. As a result, SLM is not only efficient to train, but also inherits strong capabilities already present in foundation models of different modalities.
View PDF Chat

Robust optimal subsampling based on weighted asymmetric least squares

2023-09-19
Min Ren , Shengli Zhao , Mingqiu Wang +1 more

Speech Aware Dialog System Technology Challenge (DSTC11)

2023-08-14
Hagen Soltau , Izhak Shafran , Mingqiu Wang +6 more
View PDF Chat

Heteroscedasticity identification and variable selection via multiple quantile regression

2023-08-05
Mingqiu Wang , Xiaoning Kang , Jiajuan Liang +2 more
AbstractHigh-dimensional data often display heteroscedasticity. If the heteroscedasticity is neglected in the regression model, it will produce inefficient inference for the regression coefficients. Quantile regression is not only robust to … AbstractHigh-dimensional data often display heteroscedasticity. If the heteroscedasticity is neglected in the regression model, it will produce inefficient inference for the regression coefficients. Quantile regression is not only robust to outliers, but also accommodates heteroscedasticity. This paper aims to simultaneously carry out variable selection and heteroscedasticity identification for the linear location-scale model under a unified framework. We develop a regularized multiple quantile regression approach simultaneously identifying the heteroscedasticity, seeking common features of quantile coefficients and eliminating irrelevant variables. We also establish the theoretical properties of the proposed method under some regularity conditions. Simulation studies are conducted to evaluate the finite sample performance of the proposed method, showing that it is able to identify the covariates that affect the variability of the response. We further apply the proposed method to analyse the Wage data.Keywords: Heteroscedasticitypenalty functionquantile regressionvariable selectionMathematics Subject Classifications: 62J0762F12 AcknowledgmentsThe authors would like to thank the Editor and two referees for the constructive suggestions that lead to a significant improvement over the article.Disclosure statementNo potential conflict of interest was reported by the author(s).Additional informationFundingThe research was supported by the National Natural Science Foundation of China [grant numbers 12271294, 12071248, 12071483, and 72232001], Natural Science Foundation of Liaoning Province [grant number 2022-MS-179], Department of Education of Liaoning Province [grant number LIKMZ20221565], and United International College (UIC) Start-up Research Fund [grant number R72021106], Science Foundation of Ministry of Education of China [grant number 20YJC910007].

Robust Model Structure Recovery for Ultra-High-Dimensional Varying-Coefficient Models

2023-06-16
Jing Yang , Guo‐Liang Tian , Xuewen Lu +1 more

Penalized M-Estimation Based on Standard Error Adjusted Adaptive Elastic-Net

2023-05-20
Xianjun Wu , Mingqiu Wang , Wenting Hu +2 more

MUX-PLMs: Data Multiplexing for High-throughput Language Models

2023-01-01
Vishvak Murahari , Ameet Deshpande , Carlos Jimenez-Gomez +4 more
The widespread adoption of large language models such as ChatGPT and Bard has led to unprecedented demand for these technologies. The burgeoning cost of inference for ever-increasing model sizes coupled … The widespread adoption of large language models such as ChatGPT and Bard has led to unprecedented demand for these technologies. The burgeoning cost of inference for ever-increasing model sizes coupled with hardware shortages has limited affordable access and poses a pressing need for efficiency approaches geared towards high throughput and performance. Multi-input multi-output (MIMO) algorithms such as data multiplexing, offer a promising solution with a many-fold increase in throughput by performing inference for multiple inputs at the cost of a single input. Yet these approaches are not currently performant enough to be deployed in modern systems. We change that by developing MUX-PLMs, a class of high throughput pre-trained language models (PLMs) trained with data multiplexing, that can be fine-tuned for any downstream task to yield high-throughput high-performance. Our novel multiplexing and demultiplexing modules proficiently entangle and disentangle inputs, and enable high-performance high throughput \muxplms{} that are competitive with vanilla PLMs while achieving 2x/5x inference speedup with only a $1-4\%$ drop on a broad suite of tasks.
View PDF Chat

Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding

2023-01-01
Mingqiu Wang , Izhak Shafran , Hagen Soltau +4 more
Large Language Models (LLMs) have been applied in the speech domain, often incurring a performance drop due to misaligned between speech and language representations. To bridge this gap, we propose … Large Language Models (LLMs) have been applied in the speech domain, often incurring a performance drop due to misaligned between speech and language representations. To bridge this gap, we propose a joint speech and language model (SLM) using a Speech2Text adapter, which maps speech into text token embedding space without speech information loss. Additionally, using a CTC-based blank-filtering, we can reduce the speech sequence length to that of text. In speech MultiWoz dataset (DSTC11 challenge), SLM largely improves the dialog state tracking (DST) performance (24.7% to 28.4% accuracy). Further to address errors on rare entities, we augment SLM with a Speech2Entity retriever, which uses speech to retrieve relevant entities, and then adds them to the original SLM input as a prefix. With this retrieval-augmented SLM (ReSLM), the DST performance jumps to 34.6% accuracy. Moreover, augmenting the ASR task with the dialog understanding task improves the ASR performance from 9.4% to 8.5% WER.
View PDF Chat

SLM: Bridge the thin gap between speech and text foundation models

2023-01-01
Mingqiu Wang , Wei Han , Izhak Shafran +7 more
We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM freezes the pretrained foundation … We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM freezes the pretrained foundation models to maximally preserves their capabilities, and only trains a simple adapter with just 1\% (156M) of the foundation models' parameters. This adaptation not only leads SLM to achieve strong performance on conventional tasks such as speech recognition (ASR) and speech translation (AST), but also introduces the novel capability of zero-shot instruction-following for more diverse tasks: given a speech input and a text instruction, SLM is able to perform unseen generation tasks including contextual biasing ASR using real-time context, dialog generation, speech continuation, and question answering, etc. Our approach demonstrates that the representational gap between pretrained speech and language models might be narrower than one would expect, and can be bridged by a simple adaptation mechanism. As a result, SLM is not only efficient to train, but also inherits strong capabilities already acquired in foundation models of different modalities.
View PDF Chat

AnyTOD: A Programmable Task-Oriented Dialog System

2023-01-01
Jeffrey Zhao , Yuan Cao , Raghav Gupta +6 more
Jeffrey Zhao, Yuan Cao, Raghav Gupta, Harrison Lee, Abhinav Rastogi, Mingqiu Wang, Hagen Soltau, Izhak Shafran, Yonghui Wu. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. … Jeffrey Zhao, Yuan Cao, Raghav Gupta, Harrison Lee, Abhinav Rastogi, Mingqiu Wang, Hagen Soltau, Izhak Shafran, Yonghui Wu. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
View PDF Chat

MUX-PLMs: Data Multiplexing for High-throughput Language Models

2023-01-01
Vishvak Murahari , Ameet Deshpande , Carlos Jimenez-Gomez +4 more
The widespread adoption of large language models such as ChatGPT and Bard has led to unprecedented demand for these technologies. The burgeoning cost of inference for ever-increasing model sizes coupled … The widespread adoption of large language models such as ChatGPT and Bard has led to unprecedented demand for these technologies. The burgeoning cost of inference for ever-increasing model sizes coupled with hardware shortages has limited affordable access and poses a pressing need for efficiency approaches geared towards high throughput and performance. Multi-input multi-output (MIMO) algorithms such as data multiplexing, offer a promising solution with a many-fold increase in throughput by performing inference for multiple inputs at the cost of a single input. Yet these approaches are not currently performant enough to be deployed in modern systems. We change that by developing MUX-PLMs, a class of high throughput pre-trained language models (PLMs) trained with data multiplexing, that can be fine-tuned for any downstream task to yield high-throughput high-performance. Our novel multiplexing and demultiplexing modules proficiently entangle and disentangle inputs, and enable high-performance high throughput MUX-PLMs that are competitive with vanilla PLMs while achieving 2x/5x inference speedup with only a 1-4 % drop on a broad suite of tasks.
View PDF Chat

Gemini: A Family of Highly Capable Multimodal Models

2023-01-01
Gemini Team , Rohan Anil , Sebastian Borgeaud +7 more
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano … This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models responsibly to users through services including Gemini, Gemini Advanced, Google AI Studio, and Cloud Vertex AI.
View PDF Chat

Optimal subsampling for least absolute relative error estimators with massive data

2022-07-08
Min Ren , Shengli Zhao , Mingqiu Wang

RNN Transducers for Nested Named Entity Recognition with constraints on alignment for long sequences

2022-01-01
Hagen Soltau , Izhak Shafran , Mingqiu Wang +1 more
Popular solutions to Named Entity Recognition (NER) include conditional random fields, sequence-to-sequence models, or utilizing the question-answering framework. However, they are not suitable for nested and overlapping spans with large … Popular solutions to Named Entity Recognition (NER) include conditional random fields, sequence-to-sequence models, or utilizing the question-answering framework. However, they are not suitable for nested and overlapping spans with large ontologies and for predicting the position of the entities. To fill this gap, we introduce a new model for NER task -- an RNN transducer (RNN-T). These models are trained using paired input and output sequences without explicitly specifying the alignment between them, similar to other seq-to-seq models. RNN-T models learn the alignment using a loss function that sums over all alignments. In NER tasks, however, the alignment between words and target labels are available from the human annotations. We propose a fixed alignment RNN-T model that utilizes the given alignment, while preserving the benefits of RNN-Ts such as modeling output dependencies. As a more general case, we also propose a constrained alignment model where users can specify a relaxation of the given input alignment and the model will learn an alignment within the given constraints. In other words, we propose a family of seq-to-seq models which can leverage alignments between input and target sequences when available. Through empirical experiments on a challenging real-world medical NER task with multiple nested ontologies, we demonstrate that our fixed alignment model outperforms the standard RNN-T model, improving F1-score from 0.70 to 0.74.
View PDF Chat

Description-Driven Task-Oriented Dialog Modeling

2022-01-01
Jeffrey Zhao , Raghav Gupta , Yuan Cao +6 more
Task-oriented dialogue (TOD) systems are required to identify key information from conversations for the completion of given tasks. Such information is conventionally specified in terms of intents and slots contained … Task-oriented dialogue (TOD) systems are required to identify key information from conversations for the completion of given tasks. Such information is conventionally specified in terms of intents and slots contained in task-specific ontology or schemata. Since these schemata are designed by system developers, the naming convention for slots and intents is not uniform across tasks, and may not convey their semantics effectively. This can lead to models memorizing arbitrary patterns in data, resulting in suboptimal performance and generalization. In this paper, we propose that schemata should be modified by replacing names or notations entirely with natural language descriptions. We show that a language description-driven system exhibits better understanding of task specifications, higher performance on state tracking, improved data efficiency, and effective zero-shot transfer to unseen tasks. Following this paradigm, we present a simple yet effective Description-Driven Dialog State Tracking (D3ST) model, which relies purely on schema descriptions and an "index-picking" mechanism. We demonstrate the superiority in quality, data efficiency and robustness of our approach as measured on the MultiWOZ (Budzianowski et al.,2018), SGD (Rastogi et al., 2020), and the recent SGD-X (Lee et al., 2021) benchmarks.
View PDF Chat

Unsupervised Slot Schema Induction for Task-oriented Dialog

2022-01-01
Dian Yu , Mingqiu Wang , Yuan Cao +3 more
Dian Yu, Mingqiu Wang, Yuan Cao, Izhak Shafran, Laurent Shafey, Hagen Soltau. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language … Dian Yu, Mingqiu Wang, Yuan Cao, Izhak Shafran, Laurent Shafey, Hagen Soltau. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.
View PDF Chat

Knowledge-grounded Dialog State Tracking

2022-01-01
Dian Yu , Mingqiu Wang , Yuan Cao +3 more
Knowledge (including structured knowledge such as schema and ontology, and unstructured knowledge such as web corpus) is a critical part of dialog understanding, especially for unseen tasks and domains. Traditionally, … Knowledge (including structured knowledge such as schema and ontology, and unstructured knowledge such as web corpus) is a critical part of dialog understanding, especially for unseen tasks and domains. Traditionally, such domain-specific knowledge is encoded implicitly into model parameters for the execution of downstream tasks, which makes training inefficient. In addition, such models are not easily transferable to new tasks with different schemas. In this work, we propose to perform dialog state tracking grounded on knowledge encoded externally. We query relevant knowledge of various forms based on the dialog context where such information can ground the prediction of dialog states. We demonstrate superior performance of our proposed method over strong baselines, especially in the few-shot learning setting.
View PDF Chat

Estimation and inference for transfer learning with high-dimensional quantile regression

2022-01-01
Jiayu Huang , Mingqiu Wang , Yuanshan Wu
Transfer learning has become an essential technique to exploit information from the source domain to boost performance of the target task. Despite the prevalence in high-dimensional data, heterogeneity and heavy … Transfer learning has become an essential technique to exploit information from the source domain to boost performance of the target task. Despite the prevalence in high-dimensional data, heterogeneity and heavy tails are insufficiently accounted for by current transfer learning approaches and thus may undermine the resulting performance. We propose a transfer learning procedure in the framework of high-dimensional quantile regression models to accommodate heterogeneity and heavy tails in the source and target domains. We establish error bounds of transfer learning estimator based on delicately selected transferable source domains, showing that lower error bounds can be achieved for critical selection criterion and larger sample size of source tasks. We further propose valid confidence interval and hypothesis test procedures for individual component of high-dimensional quantile regression coefficients by advocating a double transfer learning estimator, which is one-step debiased estimator for the transfer learning estimator wherein the technique of transfer learning is designed again. By adopting data-splitting technique, we advocate a transferability detection approach that guarantees to circumvent negative transfer and identify transferable sources with high probability. Simulation results demonstrate that the proposed method exhibits some favorable and compelling performances and the practical utility is further illustrated by analyzing a real example.

Speech Aware Dialog System Technology Challenge (DSTC11)

2022-01-01
Hagen Soltau , Izhak Shafran , Mingqiu Wang +6 more
Most research on task oriented dialog modeling is based on written text input. However, users interact with practical dialog systems often using speech as input. Typically, systems convert speech into … Most research on task oriented dialog modeling is based on written text input. However, users interact with practical dialog systems often using speech as input. Typically, systems convert speech into text using an Automatic Speech Recognition (ASR) system, introducing errors. Furthermore, these systems do not address the differences in written and spoken language. The research on this topic is stymied by the lack of a public corpus. Motivated by these considerations, our goal in hosting the speech-aware dialog state tracking challenge was to create a public corpus or task which can be used to investigate the performance gap between the written and spoken forms of input, develop models that could alleviate this gap, and establish whether Text-to-Speech-based (TTS) systems is a reasonable surrogate to the more-labor intensive human data collection. We created three spoken versions of the popular written-domain MultiWoz task -- (a) TTS-Verbatim: written user inputs were converted into speech waveforms using a TTS system, (b) Human-Verbatim: humans spoke the user inputs verbatim, and (c) Human-paraphrased: humans paraphrased the user inputs. Additionally, we provided different forms of ASR output to encourage wider participation from teams that may not have access to state-of-the-art ASR systems. These included ASR transcripts, word time stamps, and latent representations of the audio (audio encoder outputs). In this paper, we describe the corpus, report results from participating teams, provide preliminary analyses of their results, and summarize the current state-of-the-art in this domain.

AnyTOD: A Programmable Task-Oriented Dialog System

2022-01-01
Jeffrey Zhao , Yuan Cao , Raghav Gupta +6 more
We propose AnyTOD, an end-to-end, zero-shot task-oriented dialog (TOD) system capable of handling unseen tasks without task-specific training. We view TOD as a program executed by a language model (LM), … We propose AnyTOD, an end-to-end, zero-shot task-oriented dialog (TOD) system capable of handling unseen tasks without task-specific training. We view TOD as a program executed by a language model (LM), where program logic and ontology is provided by a designer as a schema. To enable generalization to unseen schemas and programs without prior training, AnyTOD adopts a neuro-symbolic approach. A neural LM keeps track of events occurring during a conversation and a symbolic program implementing the dialog policy is executed to recommend next actions AnyTOD should take. This approach drastically reduces data annotation and model training requirements, addressing the enduring challenge of rapidly adapting a TOD system to unseen tasks and domains. We demonstrate state-of-the-art results on STAR, ABCD and SGD benchmarks. We also demonstrate strong zero-shot transfer ability in low-resource settings, such as zero-shot on MultiWOZ. In addition, we release STARv2, an updated version of the STAR dataset with richer annotations, for benchmarking zero-shot end-to-end TOD models.

Unsupervised Slot Schema Induction for Task-oriented Dialog

2022-01-01
Dian Yu , Mingqiu Wang , Yuan Cao +3 more
Carefully-designed schemas describing how to collect and annotate dialog corpora are a prerequisite towards building task-oriented dialog systems. In practical applications, manually designing schemas can be error-prone, laborious, iterative, and … Carefully-designed schemas describing how to collect and annotate dialog corpora are a prerequisite towards building task-oriented dialog systems. In practical applications, manually designing schemas can be error-prone, laborious, iterative, and slow, especially when the schema is complicated. To alleviate this expensive and time consuming process, we propose an unsupervised approach for slot schema induction from unlabeled dialog corpora. Leveraging in-domain language models and unsupervised parsing structures, our data-driven approach extracts candidate slots without constraints, followed by coarse-to-fine clustering to induce slot types. We compare our method against several strong supervised baselines, and show significant performance improvement in slot schema induction on MultiWoz and SGD datasets. We also demonstrate the effectiveness of induced schemas on downstream applications including dialog state tracking and response generation.
View PDF Chat

Knowledge-grounded Dialog State Tracking

2022-01-01
Dian Yu , Mingqiu Wang , Yuan Cao +3 more
Knowledge (including structured knowledge such as schema and ontology and unstructured knowledge such as web corpus) is a critical part of dialog understanding, especially for unseen tasks and domains. Traditionally, … Knowledge (including structured knowledge such as schema and ontology and unstructured knowledge such as web corpus) is a critical part of dialog understanding, especially for unseen tasks and domains. Traditionally, such domain-specific knowledge is encoded implicitly into model parameters for the execution of downstream tasks, which makes training inefficient. In addition , such models are not easily transferable to new tasks with different schemas. In this work, we propose to perform dialog state tracking grounded on knowledge encoded externally. We query relevant knowledge of various forms based on the dialog context where such information can grounds the prediction of dialog states. We demonstrate superior performance of our proposed method over strong baselines, especially in the few-shot learning setting.
View PDF Chat

Doubly robust weighted composite quantile regression based on SCAD‐<i>L</i><sub>2</sub>

2021-12-20
Zhimiao Cao , Xiaoning Kang , Mingqiu Wang
In this article, a robust variable selection procedure based on the weighted composite quantile regression (WCQR) is proposed. Compared with the composite quantile regression (CQR), WCQR is robust to heavy‐tailed … In this article, a robust variable selection procedure based on the weighted composite quantile regression (WCQR) is proposed. Compared with the composite quantile regression (CQR), WCQR is robust to heavy‐tailed errors and outliers in the explanatory variables. For the choice of the weights in the WCQR, we employ a weighting scheme based on the principal component method. To select variables with grouping effect, we consider WCQR with SCAD‐ L 2 penalization. Furthermore, under some suitable assumptions, the theoretical properties, including the consistency and oracle property of the estimator, are established with a diverging number of parameters. In addition, we study the numerical performance of the proposed method in the case of ultrahigh‐dimensional data. Simulation studies and real examples are provided to demonstrate the superiority of our method over the CQR method when there are outliers in the explanatory variables and/or the random error is from a heavy‐tailed distribution.

Word-Level Confidence Estimation for RNN Transducers

2021-12-13
Mingqiu Wang , Hagen Soltau , Laurent El Shafey +1 more
Confidence estimate is an often requested feature in applications such as medical transcription where errors can impact patient care and the confidence estimate could be used to alert medical professionals … Confidence estimate is an often requested feature in applications such as medical transcription where errors can impact patient care and the confidence estimate could be used to alert medical professionals to verify potential errors in recognition. In this paper, we present a lightweight neural confidence model tailored for Automatic Speech Recognition (ASR) system with Recurrent Neural Network Transducers (RNN-T). Compared to other existing approaches, our model utilizes: (a) the time information associated with recognized words, which reduces the computational complexity, and (b) a simple and elegant trick for mapping between sub-word and word sequences. The mapping addresses the non-unique tokenization and token deletion problems while amplifying differences between confusable words. Through extensive empirical evaluations on two different long-form test sets, we demonstrate that the model achieves a performance of 0.4 Normalized Cross Entropy (NCE) and 0.05 Expected Calibration Error (ECE). It is robust across different ASR configurations, including target types (graphemes vs. morphemes), traffic conditions (streaming vs. non-streaming), and encoder types. We further discuss the importance of evaluation metrics to reflect practical applications and highlight the need for further work in improving Area Under the Curve (AUC) for Negative Precision Rate (NPV) and True Negative Rate (TNR).
View PDF Chat

A Cholesky-based sparse covariance estimation with an application to genes data

2021-05-29
Chunshi Li , Mo Yang , Mingqiu Wang +2 more
The modified Cholesky decomposition (MCD) is a powerful tool for estimating a covariance matrix. The regularization can be conveniently imposed on the linear regressions to encourage the sparsity in the … The modified Cholesky decomposition (MCD) is a powerful tool for estimating a covariance matrix. The regularization can be conveniently imposed on the linear regressions to encourage the sparsity in the estimated covariance matrix to accommodate the high-dimensional data. In this paper, we propose a Cholesky-based sparse ensemble estimate for covariance matrix by averaging a set of Cholesky factor estimates obtained from multiple variable orderings used in the MCD. The sparse estimation is enabled by encouraging the sparsity in the Cholesky factor. The theoretical consistent property is established under some regular conditions. The merits of the proposed method are illustrated through simulation and a maize genes data set.

Understanding Medical Conversations: Rich Transcription, Confidence Scores & Information Extraction.

2021-04-06
Hagen Soltau , Mingqiu Wang , Izhak Shafran +1 more
In this paper, we describe novel components for extracting clinically relevant information from medical conversations which will be available as Google APIs. We describe a transformer-based Recurrent Neural Network Transducer … In this paper, we describe novel components for extracting clinically relevant information from medical conversations which will be available as Google APIs. We describe a transformer-based Recurrent Neural Network Transducer (RNN-T) model tailored for long-form audio, which can produce rich transcriptions including speaker segmentation, speaker role labeling, punctuation and capitalization. On a representative test set, we compare performance of RNN-T models with different encoders, units and streaming constraints. Our transformer-based streaming model performs at about 20% WER on the ASR task, 6% WDER on the diarization task, 43% SER on periods, 52% SER on commas, 43% SER on question marks and 30% SER on capitalization. Our recognizer is paired with a confidence model that utilizes both acoustic and lexical features from the recognizer. The model performs at about 0.37 NCE. Finally, we describe a RNN-T based tagging model. The performance of the model depends on the ontologies, with F-scores of 0.90 for medications, 0.76 for symptoms, 0.75 for conditions, 0.76 for diagnosis, and 0.61 for treatments. While there is still room for improvement, our results suggest that these models are sufficiently accurate for practical applications.

Ensemble sparse estimation of covariance structure for exploring genetic disease data

2021-03-18
Xiaoning Kang , Mingqiu Wang

R2D2: Relational Text Decoding with Transformers

2021-01-01
Aryan Arbabi , Mingqiu Wang , Laurent El Shafey +2 more
We propose a novel framework for modeling the interaction between graphical structures and the natural language text associated with their nodes and edges. Existing approaches typically fall into two categories. … We propose a novel framework for modeling the interaction between graphical structures and the natural language text associated with their nodes and edges. Existing approaches typically fall into two categories. On group ignores the relational structure by converting them into linear sequences and then utilize the highly successful Seq2Seq models. The other side ignores the sequential nature of the text by representing them as fixed-dimensional vectors and apply graph neural networks. Both simplifications lead to information loss. Our proposed method utilizes both the graphical structure as well as the sequential nature of the texts. The input to our model is a set of text segments associated with the nodes and edges of the graph, which are then processed with a transformer encoder-decoder model, equipped with a self-attention mechanism that is aware of the graphical relations between the nodes containing the segments. This also allows us to use BERT-like models that are already trained on large amounts of text. While the proposed model has wide applications, we demonstrate its capabilities on data-to-text generation tasks. Our approach compares favorably against state-of-the-art methods in four tasks without tailoring the model architecture. We also provide an early demonstration in a novel practical application -- generating clinical notes from the medical entities mentioned during clinical visits.

Word-level confidence estimation for RNN transducers

2021-01-01
Mingqiu Wang , Hagen Soltau , Laurent El Shafey +1 more
Confidence estimate is an often requested feature in applications such as medical transcription where errors can impact patient care and the confidence estimate could be used to alert medical professionals … Confidence estimate is an often requested feature in applications such as medical transcription where errors can impact patient care and the confidence estimate could be used to alert medical professionals to verify potential errors in recognition. In this paper, we present a lightweight neural confidence model tailored for Automatic Speech Recognition (ASR) system with Recurrent Neural Network Transducers (RNN-T). Compared to other existing approaches, our model utilizes: (a) the time information associated with recognized words, which reduces the computational complexity, and (b) a simple and elegant trick for mapping between sub-word and word sequences. The mapping addresses the non-unique tokenization and token deletion problems while amplifying differences between confusable words. Through extensive empirical evaluations on two different long-form test sets, we demonstrate that the model achieves a performance of 0.4 Normalized Cross Entropy (NCE) and 0.05 Expected Calibration Error (ECE). It is robust across different ASR configurations, including target types (graphemes vs. morphemes), traffic conditions (streaming vs. non-streaming), and encoder types. We further discuss the importance of evaluation metrics to reflect practical applications and highlight the need for further work in improving Area Under the Curve (AUC) for Negative Precision Rate (NPV) and True Negative Rate (TNR).
View PDF Chat

Understanding Medical Conversations: Rich Transcription, Confidence Scores &amp; Information Extraction

2021-01-01
Hagen Soltau , Mingqiu Wang , Izhak Shafran +1 more
In this paper, we describe novel components for extracting clinically relevant information from medical conversations which will be available as Google APIs. We describe a transformer-based Recurrent Neural Network Transducer … In this paper, we describe novel components for extracting clinically relevant information from medical conversations which will be available as Google APIs. We describe a transformer-based Recurrent Neural Network Transducer (RNN-T) model tailored for long-form audio, which can produce rich transcriptions including speaker segmentation, speaker role labeling, punctuation and capitalization. On a representative test set, we compare performance of RNN-T models with different encoders, units and streaming constraints. Our transformer-based streaming model performs at about 20% WER on the ASR task, 6% WDER on the diarization task, 43% SER on periods, 52% SER on commas, 43% SER on question marks and 30% SER on capitalization. Our recognizer is paired with a confidence model that utilizes both acoustic and lexical features from the recognizer. The model performs at about 0.37 NCE. Finally, we describe a RNN-T based tagging model. The performance of the model depends on the ontologies, with F-scores of 0.90 for medications, 0.76 for symptoms, 0.75 for conditions, 0.76 for diagnosis, and 0.61 for treatments. While there is still room for improvement, our results suggest that these models are sufficiently accurate for practical applications.

A new variant of the parallel regression model with variable selection in surveys with sensitive attribute

2020-11-01
Yin Liu , Guo‐Liang Tian , Mingqiu Wang

Group selection via adjusted weighted least absolute deviation regression

2020-04-10
Xiuli Wang , Zhimiao Cao , Chao Liu +1 more

The Medical Scribe: Corpus Development and Model Performance Analyses

2020-01-01
Izhak Shafran , Nan Du , Linh Tran +7 more
There is a growing interest in creating tools to assist in clinical note generation using the audio of provider-patient encounters. Motivated by this goal and with the help of providers … There is a growing interest in creating tools to assist in clinical note generation using the audio of provider-patient encounters. Motivated by this goal and with the help of providers and medical scribes, we developed an annotation scheme to extract relevant clinical concepts. We used this annotation scheme to label a corpus of about 6k clinical encounters. This was used to train a state-of-the-art tagging model. We report ontologies, labeling results, model performances, and detailed analyses of the results. Our results show that the entities related to medications can be extracted with a relatively high accuracy of 0.90 F-score, followed by symptoms at 0.72 F-score, and conditions at 0.57 F-score. In our task, we not only identify where the symptoms are mentioned but also map them to canonical forms as they appear in the clinical notes. Of the different types of errors, in about 19-38% of the cases, we find that the model output was correct, and about 17-32% of the errors do not impact the clinical note. Taken together, the models developed in this work are more useful than the F-scores reflect, making it a promising approach for practical applications.

Statistical inference for semiparametric varying-coefficient partially linear models with a diverging number of components

2020-01-01
Mingqiu Wang , Guo‐Liang Tian , Yin Liu

Profile Statistical Inference for Partially Linear Additive Models with a Diverging Number of Parameters

2019-12-01
Xiuli Wang , Shengli Zhao , Mingqiu Wang

A Cholesky-based estimation for large-dimensional covariance matrices

2019-09-10
Xiaoning Kang , Chaoping Xie , Mingqiu Wang
This paper develops a new method to estimate a large-dimensional covariance matrix when the variables have no natural ordering among themselves. The modified Cholesky decomposition technique is used to provide … This paper develops a new method to estimate a large-dimensional covariance matrix when the variables have no natural ordering among themselves. The modified Cholesky decomposition technique is used to provide a set of estimates of the covariance matrix under multiple orderings of variables. The proposed estimator is in the form of a linear combination of these available estimates and the identity matrix. It is positive definite and applicable in large dimensions. The merits of the proposed estimator are demonstrated through the numerical study and a real data example by comparison with several existing methods.

Learning to Infer Entities, Properties and their Relations from Clinical Conversations

2019-01-01
Nan Du , Mingqiu Wang , Linh Tran +2 more
Recently we proposed the Span Attribute Tagging (SAT) Model (Du et al., 2019) to infer clinical entities (e.g., symptoms) and their properties (e.g., duration). It tackles the challenge of large … Recently we proposed the Span Attribute Tagging (SAT) Model (Du et al., 2019) to infer clinical entities (e.g., symptoms) and their properties (e.g., duration). It tackles the challenge of large label space and limited training data using a hierarchical two-stage approach that identifies the span of interest in a tagging step and assigns labels to the span in a classification step. We extend the SAT model to jointly infer not only entities and their properties but also relations between them. Most relation extraction models restrict inferring relations between tokens within a few neighboring sentences, mainly to avoid high computational complexity. In contrast, our proposed Relation-SAT (R-SAT) model is computationally efficient and can infer relations over the entire conversation, spanning an average duration of 10 minutes. We evaluate our model on a corpus of clinical conversations. When the entities are given, the R-SAT outperforms baselines in identifying relations between symptoms and their properties by about 32% (0.82 vs 0.62 F-score) and by about 50% (0.60 vs 0.41 F-score) on medications and their properties. On the more difficult task of jointly inferring entities and relations, the R-SAT model achieves a performance of 0.34 and 0.45 for symptoms and medications respectively, which is significantly better than 0.18 and 0.35 for the baseline model. The contributions of different components of the model are quantified using ablation analysis.

Learning to Infer Entities, Properties and their Relations from Clinical Conversations

2019-01-01
Nan Du , Mingqiu Wang , Linh Tran +2 more
Nan Du, Mingqiu Wang, Linh Tran, Gang Lee, Izhak Shafran. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural … Nan Du, Mingqiu Wang, Linh Tran, Gang Lee, Izhak Shafran. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
View PDF Chat

M-estimation of the mixed-type generalized linear model

2018-07-30
Ying Dong , Lixin Song , Mingqiu Wang +1 more
To investigate the features of the individual from the mixedtype model, a novel model, named the mixed-type generalized linear model, is proposed firstly in this work, which is verified to … To investigate the features of the individual from the mixedtype model, a novel model, named the mixed-type generalized linear model, is proposed firstly in this work, which is verified to be realistic and useful. We consider the robustness of M-estimation to estimate the unknown parameters of the mixed-type generalized linear model. By applying the law of large numbers and the central limit theorem, the consistency and asymptotic normality of the M-estimation for the mixed-type generalized linear model are proved with regularity assumptions. At last, in order to evaluate the finite sample performance of the estimator for the new model, several applied instances are presented, which show the good performance of the estimator.
View PDF Chat

Structure identification for varying coefficient models with measurement errors based on kernel smoothing

2018-05-14
Mingqiu Wang , Peixin Zhao , Xiaoning Kang

Adaptive group bridge estimation for high-dimensional partially linear models

2017-06-30
Xiuli Wang , Mingqiu Wang
This paper studies group selection for the partially linear model with a diverging number of parameters. We propose an adaptive group bridge method and study the consistency, convergence rate and … This paper studies group selection for the partially linear model with a diverging number of parameters. We propose an adaptive group bridge method and study the consistency, convergence rate and asymptotic distribution of the global adaptive group bridge estimator under regularity conditions. Simulation studies and a real example show the finite sample performance of our method.
View PDF Chat

Restricted profile estimation for partially linear models with large-dimensional covariates

2017-04-28
Xiuli Wang , Shengli Zhao , Mingqiu Wang

Adaptive group Lasso for high-dimensional generalized linear models

2017-02-03
Mingqiu Wang , Guo‐Liang Tian
Coauthor Papers Together
Izhak Shafran 27
Hagen Soltau 20
Laurent El Shafey 16
Yuan Cao 13
Lixin Song 10
Guo‐Liang Tian 8
Dian Yu 7
Xiaoning Kang 6
Xiuli Wang 5
Abhinav Rastogi 5
Jeffrey Zhao 5
Yonghui Wu 5
Chung‐Cheng Chiu 4
Lukáš Žilka 4
Wei Han 4
Xiaoguang Wang 4
Johan Schalkwyk 4
Shengli Zhao 4
Paul K. Rubenstein 4
Raghav Gupta 3
Linh Tran 3
Dian Yu 3
Yuan Cao 3
Nan Du 3
Ying Dong 3
Ying Xu 3
Harrison Lee 3
Nanxin Chen 3
Gang Li 3
Axel Stjerngren 2
Nithya Attaluri 2
Jeremy Wiesner 2
Balaji Lakshminarayanan 2
Shuo-Yiin Chang 2
Sanjay Ghemawat 2
Maxim Krikun 2
Vittorio Selo 2
Komal Jalan 2
Disha Shrivastava 2
Richard Tanburn 2
Nina Martin 2
Laura Knight 2
Anudhyan Boral 2
Jessica Landon 2
Arthur Guez 2
Devendra Singh Sachan 2
Vincent J. Hellendoorn 2
Josip Djolonga 2
Ricardo Aguilar 2
Zhimiao Cao 2