Multinomial logistic regression is a generalization of binary logistic regression to multiclass problems. This note will explain the nice geometry of the likelihood function in estimating the model parameters by looking at the Hessian of the MLR objective function.

This note describes the EM algorithm for estimating a Dirichlet distribution. One result that I found particularly interesting is the posterior distributions of gamma-distributed random variables given the observation of their corresponding Dirichlet distribution are also gamma-distributed.

This note collects a set of inequalities for smooth convex functions. As an introduction, we shall prove the convergence of gradient descent method using some of these inequalities. In particular, we show that gradient descent with fixed step size converges to a global minimum at a sublinear rate when the objective is smooth convex, and at a linear rate when strong convexity is added. It is also surprising to me that while the loose bound, $1-\mu/L$, is commonly used in standard convex analysis, the tight bound, $(1-\mu/L)^2$, is a lesser known fact and can hardly be found in notes/lectures on this topic.

We will study the proof of convergence of two well-known acceleration techniques - Heavy-Ball Method and Nesterov’s Accelerated Gradient on minimizing a convex, quadratic function.

In this note we will study matrix perturbation theory and find out the answer to some basic questions such as what happens when adding small perturbations to a symmetric matrix, or how much the invariant subspace spanned by its eigenvectors can change. Understanding the effect of small perturbation on matrices is the key to analysis of local convergence in many optimization algorithms.

As a follow-up from my previous note on convex optimization, this note studies the so-called projected gradient descent method and its sibling, proximal gradient descent. Using the fundamental inequalities from convex analysis, we shall show that both of the methods enjoy similar convergence properties to gradient descent for unconstrained optimization.

publications

Fuzzy linguistic propositional logic based on refined hedge algebra

Tran Duc-Khanh, Vu Viet-Trung, Doan The-Vinh and Nguyen Minh-Tam

IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), 2013

@inproceedings{tran2013fuzzy, title={Fuzzy linguistic propositional logic based on refined hedge algebra}, author={Tran, Duc-Khanh and Vu, Viet-Trung and Doan, The-Vinh and Nguyen, Minh-Tam}, booktitle={2013 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)}, pages={1--8}, year={2013}, organization={IEEE} }

Resolution in linguistic first order logic based on linear symmetrical hedge algebra

Thi-Minh-Tam Nguyen, Vu Viet-Trung, Tran Duc-Khanh and The-Vinh Doan

International Conference on Information Processing and Management of Uncertainty (IPMU) in Knowledge-Based Systems, 2014

@inproceedings{vu2014resolution, title={Resolution in linguistic first order logic based on linear symmetrical hedge algebra}, author={Thi-Minh-Tam Nguyen and Vu, Viet-Trung and Tran, Duc-Khanh and The-Vinh Doan}, booktitle={International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems}, pages={345--354}, year={2014}, organization={Springer} }

Resolution in Linguistic Propositional Logic Based on Linear Symmetrical Hedge Algebra

Thi-Minh-Tam Nguyen, Vu Viet-Trung, Tran Duc-Khanh and The-Vinh Doan

International Conference on Knowledge and Systems Engineering (KSE), 2014

@inproceedings{vu2014resolution, title={Resolution in linguistic propositional logic based on linear symmetrical hedge algebra}, author={Thi-Minh-Tam Nguyen and Vu, Viet-Trung and Tran, Duc-Khanh and The-Vinh Doan}, booktitle={Knowledge and Systems Engineering}, pages={327--338}, year={2014}, organization={Springer} }

Adaptive Step Size Momentum Method for Deconvolution

Trung Vu and Raviv Raich

IEEE Statistical Signal Processing (SSP) Workshop, 2018

@inproceedings{vu2019local, title={Local Convergence of the {H}eavy {B}all Method in Iterative Hard Thresholding for Low-rank Matrix Completion}, author={Vu, Trung and Raich, Raviv}, booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, pages={3417--3421}, year={2019}, organization={IEEE} }

Accelerating Iterative Hard Thresholding for Low-Rank Matrix Completion via Adaptive Restart

Trung Vu and Raviv Raich

IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), 2019

@inproceedings{vu2019accelerating, title={Accelerating Iterative Hard Thresholding for Low-Rank Matrix Completion via Adaptive Restart}, author={Vu, Trung and Raich, Raviv}, booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, pages={2917--2921}, year={2019}, organization={IEEE} }

On Convergence of Projected Gradient Descent for Minimizing a Large-scale Quadratic over the Unit Sphere

Student Paper Award (2nd place)

Trung Vu, Raviv Raich and Xiao Fu

IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2019

@inproceedings{vu2019convergence, title={On Convergence of Projected Gradient Descent for Minimizing a Large-Scale Quadratic Over the Unit Sphere}, author={Vu, Trung and Raich, Raviv and Xiao, Fu}, booktitle={IEEE International Workshop on Machine Learning for Signal Processing (MLSP)}, pages={1--6}, year={2019}, organization={IEEE} }

A Novel Attribute-based Symmetric Multiple Instance Learning for Histopathological Image Analysis

Trung Vu, Phung Lai, Raviv Raich, Anh Pham, Xiaoli Z. Fern and UK Arvind Rao

@article{vu2020novel, title={A Novel Attribute-based Symmetric Multiple Instance Learning for Histopathological Image Analysis}, author={Vu, Trung and Lai, Phung and Raich, Raviv and Pham, Anh and Fern, Xiaoli Z and Rao, UK Arvind}, journal={IEEE Transactions on Medical Imaging}, volume={39}, number={10}, pages={3125--3136}, year={2020}, publisher={IEEE} }

Exact Linear Convergence Rate Analysis for Low-Rank Symmetric Matrix Completion via Gradient Descent

Trung Vu and Raviv Raich

IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), 2021

@inproceedings{vu2021exact, title={Exact Linear Convergence Rate Analysis for Low-Rank Symmetric Matrix Completion via Gradient Descent}, author={Vu, Trung and Raich, Raviv}, booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, pages={3240--3244}, year={2021}, organization={IEEE} }

Perturbation Expansions and Error Bounds for the Truncated Singular Value Decomposition

@article{vu2021perturbation, title={Perturbation Expansions and Error Bounds for the Truncated Singular Value Decomposition}, author={Vu, Trung and Chunikhina, Evgenia and Raich, Raviv}, journal={Linear Algebra and its Applications}, year={2021}, publisher={Elsevier} }

A Closed-Form Bound on the Asymptotic Linear Convergence of Iterative Methods via Fixed Point Analysis

@article{vu2021closed, title={A Closed-Form Bound on the Asymptotic Linear Convergence of Iterative Methods via Fixed Point Analysis}, author={Vu, Trung and Raich, Raviv}, journal={Optimization Letters}, year={2022}, publisher={Elsevier} }

On Local Linear Convergence of Projected Gradient Descent for Unit-Modulus Least Squares

@article{vu2020novel, title={On Asymptotic Linear Convergence of Projected Gradient Descent for Constrained Least Squares}, author={Vu, Trung and Raich, Raviv}, journal={IEEE Transactions on Signal Processing}, volume={70}, pages={4061--4076}, year={2022}, publisher={IEEE} }

On Asymptotic Linear Convergence Rate of Iterative Hard Thresholding for Matrix Completion

@article{vu2020novel, title={On Local Linear Convergence Rate of Iterative Hard Thresholding for Matrix Completion}, author={Vu, Trung and Chunikhina, Evgenia and Raich, Raviv}, journal={IEEE Transactions on Signal Processing}, volume={70}, pages={5940--5953}, year={2022}, publisher={IEEE} }

teaching

CS 271 - Computer Architecture and Assembly Language

Online Course, EECS, Oregon State University - Fall 2016, Fall 2019

CS 290 - Web Development

Online Course, EECS, Oregon State University - Spring 2019, Summer 2019, Fall 2020, Winter 2021, Spring 2021, Fall 2021

CS 290 - Web Development (Head TA)

Online Course, EECS, Oregon State University - Spring 2020, Summer 2020

CS 362 - Software Engineering II

Online Course, EECS, Oregon State University - Summer 2021

CS 261 - Data Structures

Online Course, EECS, Oregon State University - Winter 2019

ECE464/564 - Digital Signal Processing

Online Course, EECS, Oregon State University - Winter 2020