非参数可加模型的迭代自适应稳健变量选择.pdf

应用概率统计第 40 卷第 2 期2024 年 4 月Chinese Journal of Applied Probability and StatisticsApr.,2024,Vol.40,No.2,pp.201-228doi:10.3969/j.issn.1001-4268.2024.02.001Iterative Adaptive Robust Variable Selection inNomparametric Additive ModelsZHU Nenghui(School of Mathematics and Statistics,Xiamen University of Technology,Xiamen,361024,China)YOU Jinhong(School of Statistics and Management,Shanghai University of Finance and Economics,Shanghai,200433,China)XU Qunfang(Business School,Ningbo University,Ningbo,315211,China)Abstract:By utilizing the robust loss function,B-spline approximation and adaptive group Las-so,a nonparametric additive model is investigated to identify insignificant covariates for the“largep small n”setting.Compared with the ordinary least-square adaptive group Lasso,the proposedmethod is resistant to heavy-tailed errors or outlines in the responses.To prove facilitate presen-tation,a more general weighted robust group Lasso estimator is considered.Moreover,the weightvectors play a pivotal role for the suggested estimators to enjoy the model selection oracle propertyand asymptotic normality.The robust group Lasso and adaptive robust group Lasso can be seenas special circumstances of different weight vectors.In practice,we use the robust group Lasso toobtain an initial estimator to reduce the dimension of the problem,and then apply the iterativeadaptive robust group Lasso to select nonzero components.The results of simulation studies showthat the proposed methods work well with samples of moderate size.A high-dimensional geneTRIM32 data is used to illustrate the application of the proposed method.Keywords:adaptive group Lasso;high-dimensional data;nonparametric regression;oracle prop-erty;robust estimation2020 Mathematics Subject Classification:primary 62G35;secondary 62A01Citation:ZHU N H,YOU J H,XU Q F.Iterative adaptive robust variable selection in nom-parametric additive modelsJ.Chinese J Appl Probab Statist,2024,40(2):201228.The project was supported by the National Natural Science Foundation of China(Grant No.11971291),the National Social Science Foundation of China(Grant No.19BTJ032),Fujian Alliance of Mathematics(Grant No.2023SXLMMS10)and Scientific Research Climbing Program of Xiamen University of Tech-nology(Grant No.XPDKT20037).Corresponding author,E-mail:.Received October 30,2023.Revised January 9,2024.202Chinese Journal of Applied Probability and StatisticsVol.401IntroductionConsider the following nonparametric additive model:Yi=pj=1fj(Xij)+i,(1)where(Yi,Xi)are independent and identically distributed(i.i.d.)with the same distribu-tion as(Y,X),X=(X1,X2,Xp)Tis the p-dimensional predictor and iis the noisewith mean zero,the fjs are unknown functions.The statistical problem is to determinewhich additive components are insignificant components(i.e.,fj 0)and estimate thenonzero components under the heavy-tailed errors or outlines in responses.The chal-lenge becomes more serious when the number of covariates diverges with the sample size.Therefore,we present a robust group Lasso penalization-based procedure to identify in-significant covariates for the“large p small n”setting under the heavy-tailed errors oroutlines in responses,and show that the proposed method can correctly select the nonzerocomponents with high probability.There is a large body of literature on penalized methods for variable selection andestimation with high-dimensional data.Penalized methods have been proposed includingLasso1,SCAD penalty2,3,MCP4,etc.Particularly in high-dimensional settings,muchprogress has been made in the variable selection,estimation and prediction propertiesof the Lasso,see 419,among others.All these works are assumed a linear or otherparametric model.However,in many applications,nonparametric models is more flexiblethan the linear model and can be fitted to high-dimensional data when fully nonparametricmodels is infeasible due to“curse of dimensionality”.As we known,the additive model is a widely used nonparametric models.However,in practise,only some of the components are really nonparametric,and other compo-nents contain predictors unrelated to the responses.For this,several independent workshave shown that the sparse additive models can be fitted successfully to various data setswith large dimensions.Combining ideas from sparse linear modeling and additive non-parametric regression,Ravikumar et al.20showed that SPAM can be effective in fittingsparse nonparametric models in high dimensional data.Meier et al.11proposed a newsparsity-smoothness penalty for high-dimensional generalized additive models.Using B-spline bases,Huang et al.21applied group Lasso iteratively for the additive model andshowed that adaptive group Lasso can be used for high-dimensional nonparametric prob-lems despite that the initial group Lasso estimator cannot achieven consistency,thusNo.2ZHU N.H.,et al.:Iterative Adaptive Robust Variable Selection in Nomparametric Additive Models203extending the results of 7.Similar to the ideas of 22 and 12,Lian et al.23proposed adouble penalization based procedure to distinguish covariates that enter the nonparamet-ric and parametric parts and to identify insignificant covariates simultaneously the“largep small n”setting.However,the above mentioned estimations are mostly built on least squares(LS)typemethods.LS-type method may suffer from much bias when the error follows a heavy-taileddistribution or in the presence of outliers.To overcome this drawback,in this paper,aniterative robust group Lasso is proposed for nonparametric additive model,in which thenumber of additive components may be larger than the sample size.The idea behindour method is similar to those of 21 and 15.We considered a nonparametric additivemodel in the“large p small n”setting and the number of nonzero fjs is assumed fixed.Similar as 21,by utilizing B-splines to approximate the unknown functions fj(x),theproblem of component selection becomes that of selecting the groups of coefficients in theexpansion.Whereas,being different from 21,we introduce the robust loss function toadaptive group Lasso penalty.Compared with the ordinary least-square adaptive groupLasso21,the adaptive robust group Lasso is resistant to heavy-tailed errors or outlinesin the responses.Furthermore,to prove facilitate presentation,we consider more gener-ally the weighted robust group Lasso(WR-Group)estimator with nonnegative weightsd1,d2,dp.Moreover,we show that the choice of the weight vector d plays a pivotalrole for the WR-Group Lasso estimate to enjoy the model selection oracle property andasymptotic normality.The robust group Lasso and adaptive robust group Lasso can beseen as special circumstances of weighted robust group Lasso by using different weight.In practice,we use the robust group Lasso to obtain an initial estimator to reduce thedimension of the problem,and then apply the iterative adaptive robust group Lasso toselect nonzero components.Under appropriate conditions,the iterative adaptive robustgroup Lasso possessed the oracle property in the sense that it is as efficient as the estimatorwhen the true model is known prior to statistical analysis.The paper is organized as follows.In Section 2,we first propose a class of variableselection procedures for identifying the insignificant components via the iterative adap-tive robust group Lasso penalized approach and then study the statistical properties ofthe proposed procedures.In Section 3,simulations studies are carried out to assess theperformance of the proposed method and we also apply the method to some real data asillustrations in Section 4.Section 5 states the concluding remarks.The proofs for themain theoretical results are provided in the Appendix A.204Chinese Journal of Applied Probability and StatisticsVol.402Iterative Additive Robust Group Lasso in Additive Models2.1The Spline EstimatorIn this paper,we will use a2=(Ki=1|a|2)1/2to denote the l2(Euclidean)norm of avector a RK,(A)is the eigenvalues of matrix A,min(A)and max(A)are the smallestand largest eigenvalues of matrix A.Besides,we abbreviate pnas p for simplicity.We use B-spline to approximate the components.Without loss of generality,we sup-pose the distribution of Xjtakes value on 0,1,and also impose the condition Efj(Xj)=0 for identifiability.Let 0=0 1 K 0;,enj2=0.Qn(n,n2)=ni=1(Yi(Xi)Tn)+nn2pj=1wnjnj2.(4)Here,we define 0=0.Thus,the components not selected by the R-GLE arenot included in Step 2.The adaptive robust group Lasso estimator(AR-GLE)isbn=argminnQn(n,n2).Finally,the AR-GLE of fjarebfnj(x)=Kk=1bjkBjk(x),1 6 j 6 p.To prove facilitate presentation,we consider more generally the weighted robust groupLasso estimator with nonnegative weights d1,d2,dp.LetLn(n,n)=ni=1(Yi(Xi)Tn)+nnpj=1djnj2.(5)206Chinese Journal of Applied Probability and StatisticsVol.40The minimizers in(3)and(4)are particular case of(5)and corresponds to the case withdj 1 and dj=wnj.Now we call the estimators of(5)as weighted robust group Lassoestimators(WR-GLE).2.3Asymptotic Results2.3.1AssumptionIn(1),without loss of generality,suppose that the first components are nonzero,thatis,fj(x)=0,1 6 j 6 s,but fj(x)0,s+1 6 j 6 p.Let A1=1,2,s andA0=s+1,s+2,p.Define f2=10f2(x)dx for any function f,whenever theintegral exists.Let|B|denote the cardinality of any set B 1,2,p.Throughoutthis paper,c,C denotes a generic constants that might assume different values at differentplaces.We make the following assumptions:(A1)The number of nonzero components s is fixed.Let 1,2,nbe independentrandom noises with values in some space.(A2)The covariate vector X has a continuous density supported on 0,1p.Furthermore,the marginal densities of Xj(1 6 j 6 p)are all bounded from below and above bytwo fixed positive constants,respectively.(A3)Efj(Xj)=0,1 6 j 6 s,and fj 0 for j s.(A4)For fj,1 6 j 6 p satisfies a Holder condition of order 1 0such that for1 6 j 6 pfj(x)=Kk=1jkBjk(x)+Rj(x)=BTjj+Rj(x),(6)supt0,1|Rj(x)|6 CK,j=1,2,p,(7)wherej=(j1,j2,jK)T,Bj=(Bj1(x),Bj2(x),BjK(x)T.No.2ZHU N.H.,et al.:Iterative Adaptive Robust Variable Selection in Nomparametric Additive Models207Define =and the discontinuities of()as.the following assumptions on and are required for our asymptotic results.(A5)Define:(Xi)Tn,yi)=(yi(Xi)Tn),we assume(s,y)that satisfying aLipschitz condition:there exists C 1,such that|(s1,y)(s2,y)|6 C|s1 s2|.(A6)F()=0(F is cumulative distribution function of i).E(1)=0,0 E2(1)ssupNK?1nZTjZA1?2,0,define the-covering number N(,n)as the minimumnumber of balls with radius necessary to cover the class.The following assumption isabout the N(,n):(A8)If n6 Rn(0 Rn 0,such thatln(1+N(2sRn,n)6 A22s,0 6 s 6 S,where S:=mins 1:2s6 4/n.2.3.2Oracle Regularized EstimatorTo evaluate newly proposed method,we first study how well one can do with theassistance of the oracle information on the locations of signal covariates.Then,we usethis to establish the asymptotic property of our estimator without the oracle assistance.Denote byeon=(eoTA1,OT)Tthe oracle regularized estimator(ORE)witheoA1 RsKand Obeing the vector of all zeros,which minimizes Ln(n,n)over the space n=(TA1,TA0)T RpK:TA0=OT R(ps)K.The next theorem shows that ORE is consistent,and208Chinese Journal of Applied Probability and StatisticsVol.40estimates the correct sign of the true coefficient vector with probability tending to one.We use d1to denote the first s elements of d.Foren=(eTn1,eTn2,eTnp)T,n=(Tn1,Tn2,Tnp)T.We sayen=0nifsgn(enj2)=sgn(nj2),1 6 j 6 p,where sgn(|x|)=1 if|x|0 else sgn(|x|)=0.Theorem 3(Oracle Property)Suppose that(A1)to(A5)hold,d12=O(1nK ln(pK/n),n Cln(pK)/n,then(i)IfK2ln(pK)/n 0,ln(pK)/n=O(K2),then with probability at least1 O(pK)cs),pj=1eonj nj22=Op(K2ln(pK)n)+O(1K21)+O(K22n)=Op(rn),wherern=K2ln(pK)/n+1/K21+K22n.(ii)If in additionalr1nmin16j6snj2,thenP(eon=0n)1.Remark 4As show in Theorem 3,ifn=ln(pK)/n,thend12=O(K)andthe consistency ofeonin term of the vectorl2-norm is given by ratern=K2ln(pK)/n.Part(i)implies that the difference between the coefficients in the spline representation of thenonparametric function(1)and WR-GL oracle estimators converges to zero in probability.The rate of convergence is determined by three terms:the stochastic error in estimating thenonparametric components(the first term),the spline approximation error(the second term)and the bias due to penalization(the third term).2.3.3Weighted Robust Group Lasso EstimatorIn the following theorem,we show that even without the oracle information,WR-GLE enjoys the same asymptotic property as in Theorem 3 while the weighted vector isappropriately chosen under the assumption(A8).Theorem 5Suppose that(A1)to(A8)and the conditions of Theorem 3 hold,inaddition,assume thatminjA0dj=O(r1n),d12=O(1nK ln(pK)/n)andAlogn2=o(nn).Definernas in Theorem 3.Then with probability at least1 O(pK)cs),thereexist a global minimizerbn=(bTnA1,bTnA0)TofLn(n,n)which satisfies(i)bTnA0=O;(ii)pj=1bnj nj22=Op(rn).No.2ZHU N.H.,et al.:Iterative Adaptive Robust Variable Selection in Nomparametric Additive Models209Remark 6Ifn=O(ln(pK)/n),thenA logn2=O(ln(pK).Under theweight vector employing the condition:minjA0dj=O(r1n)andd12=O(K),Theorem5 holds.Thus,under the conditions of Theorem 5,we have seen that the choice of the weightvectordplay a pivotal role for the WR-GLE to enjoy the model selection oracle property.Theorem 7Suppose that(A1)to(A8)hold andn Cln(pK)/n.Further-more,assume thatminjA0dj=O(r1n),d12=O(1nK ln(pK)/n)andA logn2=o(nn),then with probability at least1 O(pK)cs)(i)All the nonzero additive componentsfj(1 6 j 6 s)are selected ifK ln(pK)/n 0,ln(pK)/n=O(K2).(ii)bfnj fnj22=Op(K ln(pK)/n)+O(1/K2)+O(K2n),j A1.Remark 8Ifn=O(ln(pK)/n),thenA logn2=O(ln(pK).Under theweight vector employing the condition:minjA0dj=O(r1n)andd12=O(K),theweighted robust group Lasso selects all the nonzero additive components with high probabilityand present the convergence rateK ln(pK)/nof the nonparametric components.2.4Properties of the Adaptive Robust Group LassoIn previous sections,we have seen that the choice of the weight vector d plays apivotal role for the WR-GLE to enjoy the model selection oracle property and asymptoticnormality.In fact,conditions in Theorem 5 require that minjA0dj=O(r1n)that d12does not diverge too fast.For robust group Lasso,d12=s,then these conditionsbecome very restrictive.Hence,an adaptive choice of weights is needed to ensure thatthose conditions are satisfied.To this end,we propose a two-step procedure similar as21.First,we use following robust group Lasso estimators as the initial estimateeini.The following Theorem 9 is same as the proofs in Theorem 1 of 27,so we omit the proof.Theorem 9Suppose that the conditions in supplemental Appendix B of Theorem 1of 27 hold,K=O(n1/(2+1)and letm0=(pK)(nln(n pK)q22)andn1=nln(n pK)(m0+sK)q,whereab:=mina,b,ab:=maxa,b,and pis a sequence of positive numbers,possibly data-dependent.Furthermore,we assume thatn1K1/2(qn)1 0.We have(a)The estimated spline coefficient vector satisfiese rn=eini 2.pn1sK 0.210Chinese Journal of Applied Probability and Statisti