BEV-radar:毫米波雷达-相机双向融合的三维目标检测.pdf
《BEV-radar:毫米波雷达-相机双向融合的三维目标检测.pdf》由会员分享,可在线阅读,更多相关《BEV-radar:毫米波雷达-相机双向融合的三维目标检测.pdf(10页珍藏版)》请在咨信网上搜索。
1、BEV-radar:bidirectional radar-camera fusion for 3D objectdetectionYuanZhao1,LuZhang2,JiajunDeng3,andYanyongZhang11School of Computer Science and Technology,University of Science and Technology of China,Hefei 230027,China;2Institute of Artificial Intelligence,Hefei Comprehensive National Science Cent
2、er,Hefei 230088,China;3Department of Electrical Engineering,University of Sydney,NSW 2006,AustraliaCorrespondence:YanyongZhang,E-mail:2024TheAuthor(s).ThisisanopenaccessarticleundertheCCBY-NC-ND4.0license(http:/creativecommons.org/licenses/by-nc-nd/4.0/).Cite This:JUSTC,2024,54(1):0101(8pp)ReadOnlin
3、eAbstract:ExploringmillimeterwaveradardataascomplementarytoRGBimagesforameliorating3Dobjectdetectionhasbecomeanemergingtrendforautonomousdrivingsystems.However,existingradar-camerafusionmethodsarehighlydependentonthepriorcameradetectionresults,renderingtheoverallperformanceunsatisfactory.Inthispaper
4、,wepro-poseabidirectionalfusionschemeinthebird-eyeview(BEV-radar),whichisindependentofpriorcameradetectionres-ults.Leveragingfeaturesfrombothmodalities,ourmethoddesignsabidirectionalattention-basedfusionstrategy.Spe-cifically,followingBEV-based3Ddetectionmethods,ourmethodengagesabidirectionaltransfo
5、rmertoembedinforma-tionfrombothmodalitiesandenforcesthelocalspatialrelationshipaccordingtosubsequentconvolutionblocks.Afterem-beddingthefeatures,theBEVfeaturesaredecodedinthe3Dobjectpredictionhead.WeevaluateourmethodonthenuS-cenesdataset,achieving48.2mAPand57.6NDS.Theresultshowsconsiderableimproveme
6、ntscomparedtothecamera-onlybaseline,especiallyintermsofvelocityprediction.Thecodeisavailableathttps:/ number:TP399Document code:A1 IntroductionThe perception system in autonomous driving is usuallyequipped with different types of sensors.Complementarymulti-modalsensorsavoidunexpectedrisksbuttakeonne
7、wchallengeswhilesensorfusion.Recentworkshavefocusedonvisualsensors1,typicallyprovidingdenseandredundantinformation.However,visualsensorsareusuallynotstableenoughforadverseweatherconditions(i.e.,rain,snow,andfog).Inadditiontothehighcost,thefusionofvisualsensorscannot fully sustain the perception syst
8、em in variableautonomousscenarios,whichrequiresrobustness.Aside from LiDAR and cameras,radar has aslo beenwidely used in autonomous scenes for speed measurementand auxiliary location prediction,but rarely in visual tasksduetoitsphysicalnature.Whilestabilityandpenetrationbe-nefitfromtheirphysicalprop
9、erties,sparseresults,noisyfea-tures,andlackofverticalinformationarecrucialproblemsbrought by frequently-used automotive radar.Randomlyscatteredsignalsamongvehicles,buildings,andobstaclesob-tainhighspecularreflectivityandmulti-patheffects.Whilethecomplementarycharacteristicsofcameraandradarareef-fect
10、ive,thefusionstrategyfacesseveralchallenges.First,theresultsofthemm-waveradarprojectedontheimageviewonlyhavedirectionandrange,whichdoesnotprovidevertic-alinformationandleadstosomebiaswhenprojectedonthecameraview.Moreover,theimagecannotrelymerelyontheprojected radar depth,as multi-path effectivity pr
11、oducesinaccurateresultsforradardetection.Compared to the richer and more accurate informationprovidedbyvisualsensors,thealignmentoffeaturesbetweenthecameraandtheradarisachallengingproblem.Withoutverticalinformation,somemethods2,3rectifytheverticaldir-ectioninthefrontviewafterprojectingradarpointstoi
12、mageplanes.Higherperformanceleveragesonfirst-stagepropos-als from the camera and then constructs a soft associationbetweenobjectsandfeaturesaccordingtotheextrinsicmat-rix,asshowninFig.1.Insteadofassociationmethods,trans-formingbothfeaturestobird-eyeviews(BEV)canextremelyrelievetheproblem,concerningt
13、wokeypoints:amorecom-patibledecoupledfusionstrategyforradardataandabetterpromotionforbothmodalities.InspiredbyBEVfusionmethods4,5,weimplementBEV-radar,anend-to-endfusionapproachforradarandcameras,whichcanbeconvenientlyusedforotherBEVsforcamerabaselines.Beforefusion,radarencodersareusedforpillarex-tr
14、actionandtensorcompaction.BEV-radarfocusesoninsert-ingdenseradartensorsintotheBEVimagefeaturesgener-ated by the camera baseline.Bidirectionally,radar featuresandimagefeaturesarepromotedtotheirrespectivedecodersaccording to cross-attention.Despite the simplicity of thebasicidea,theevaluationonthenuSc
15、enesdatasetperformsoutstandingresultsinthe3Dobjectdetectionbenchmarks.Itachievesanimprovementoverthecamera-onlybaselinesandperforms well even compared to other radar-camera fusionArticlehttp:/Received:January 15,2023;Accepted:April 03,202301011DOI:10.52396/JUSTC-2023-0006JUSTC,2024,54(1):0101studies
16、.Besides,fortheoriginalintentionoftheexperiment,radarfusionbehavesstablywith+10%mAPand+15%NDSboostinadversescenes.Wemakethefollowingcontributions:(I)Weconstructanend-to-endBEVframeworkforradarandcamerafusion.Insteadofrelyingonthefirst-stagedetec-tionresultsprovidedbythecamera,thisintegralnetworkin-s
17、tructs a portable and robust type that does not dependstronglyonthecamera.(II)Weproposeanovelbidirectionalfusionstrategycom-paredtovanillacrossattention,whichissuitableformulti-modalfeatureswithspatialrelationships.Itperformseffect-ivelydespitethehugediversityofradarandcameras.(III)Weachieveacompara
18、tivecamera-radar3DdetectionperformanceonthenuScenesdataset.Comparedtoasinglemodality,wesolvethedifficultproblemofvelocitypredic-tion,whichisnon-trivialinautonomous.2 Materials and methods2.1 Related workCamera-only 3D detection.Monocular 3D detectionrequirestheestimationof3Dboundingboxeswhileusingam
19、onocularcamera.Thekeyquestionishowtoregressthedepthinformationonthe2Dview.Earlierworksreliedon2Ddetectionnetworkswithadditionalsub-networksfor3Dpro-jection6,7.SeveralworkshaveattemptedtoconvertRGBin-formationinto3Drepresentations,suchaspseudo-LiDAR8,9andorthographicfeaturetransform10.Severalstudies1
20、1intro-ducedkeypointdetectionforcentersandused2Dobjectde-tection prediction as regression auxiliary.In recent works,camera-onlymethodsdirectlypredictedresultson3DspacesorBEVfeatures5,12,13.TheyoperateddirectlyontheBEVfea-tures transformed from the front view according tocalibration.Camera-fusion 3D
21、detection.Thekeypointoftheassoci-ationmodalityfusionmethodsistofindtheinterrelatedspa-tialrelationshipsamongmulti-modalsensors.Inrecentyears,fusionapproacheshavemainlyfocusedonLiDARandcam-eras.Someearlierworks14,15mappedthedatafrommulti-viewsintounifiedtypeslikeimageorBEV.Pointpainting1creat-ivelypr
22、oposesthesegmentationofinformationfromimagesontopointcloud.Duetothesensitivitytoadverseweatherconditions,MVDNet16firstdesignedafusedarea-wisenet-workforradarandLiDARinafoggysimulatedenvironment.MotivatedbythecostofLiDAR,Ref.17researchedtheim-provementoffusionontinyobjectswithcameraandradar,andRef.18
23、introducedthetransformerforfeature-levelfu-sion.However,the 2D convolution of the projected radarpoints comes with useless computations and does not takeintoaccountthesparsityoftheradar.Restrictedbythefrontview,spatialrelationshipsbetweendifferentmodalitiesrelyontheresultspredictedduringthefirststag
24、e.Bytransform-ingfeaturesfromtheirrespectiveviewstoaunifiedBEV,BEVFusion4predictedthedepthprobabilitiesforimagefea-turesandprojectedthepseudo-3DfeaturestotheBEVbasedontheirextrinsicparameters.Transfusion19compressedcam-erafeaturesalongtheverticalaxistoinitializetheguidingqueryandthealignresultsofthe
25、firststagebacktoimageplanes.2.2 ApproachInthiswork,wepresentBEV-radar,aradar-camerafusionImage Backbone Image Backbone Radar backbone Radar backbone Radar backbone Image backbone Radar backbone Regression HeadsRegression HeadsRegression HeadsRegression headsPredictionAssociationImage Backbone Image
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- BEV radar 毫米波 雷达 相机 双向 融合 三维 目标 检测
1、咨信平台为文档C2C交易模式,即用户上传的文档直接被用户下载,收益归上传人(含作者)所有;本站仅是提供信息存储空间和展示预览,仅对用户上传内容的表现方式做保护处理,对上载内容不做任何修改或编辑。所展示的作品文档包括内容和图片全部来源于网络用户和作者上传投稿,我们不确定上传用户享有完全著作权,根据《信息网络传播权保护条例》,如果侵犯了您的版权、权益或隐私,请联系我们,核实后会尽快下架及时删除,并可随时和客服了解处理情况,尊重保护知识产权我们共同努力。
2、文档的总页数、文档格式和文档大小以系统显示为准(内容中显示的页数不一定正确),网站客服只以系统显示的页数、文件格式、文档大小作为仲裁依据,平台无法对文档的真实性、完整性、权威性、准确性、专业性及其观点立场做任何保证或承诺,下载前须认真查看,确认无误后再购买,务必慎重购买;若有违法违纪将进行移交司法处理,若涉侵权平台将进行基本处罚并下架。
3、本站所有内容均由用户上传,付费前请自行鉴别,如您付费,意味着您已接受本站规则且自行承担风险,本站不进行额外附加服务,虚拟产品一经售出概不退款(未进行购买下载可退充值款),文档一经付费(服务费)、不意味着购买了该文档的版权,仅供个人/单位学习、研究之用,不得用于商业用途,未经授权,严禁复制、发行、汇编、翻译或者网络传播等,侵权必究。
4、如你看到网页展示的文档有www.zixin.com.cn水印,是因预览和防盗链等技术需要对页面进行转换压缩成图而已,我们并不对上传的文档进行任何编辑或修改,文档下载后都不会有水印标识(原文档上传前个别存留的除外),下载后原文更清晰;试题试卷类文档,如果标题没有明确说明有答案则都视为没有答案,请知晓;PPT和DOC文档可被视为“模板”,允许上传人保留章节、目录结构的情况下删减部份的内容;PDF文档不管是原文档转换或图片扫描而得,本站不作要求视为允许,下载前自行私信或留言给上传者【自信****多点】。
5、本文档所展示的图片、画像、字体、音乐的版权可能需版权方额外授权,请谨慎使用;网站提供的党政主题相关内容(国旗、国徽、党徽--等)目的在于配合国家政策宣传,仅限个人学习分享使用,禁止用于任何广告和商用目的。
6、文档遇到问题,请及时私信或留言给本站上传会员【自信****多点】,需本站解决可联系【 微信客服】、【 QQ客服】,若有其他问题请点击或扫码反馈【 服务填表】;文档侵犯商业秘密、侵犯著作权、侵犯人身权等,请点击“【 版权申诉】”(推荐),意见反馈和侵权处理邮箱:1219186828@qq.com;也可以拔打客服电话:4008-655-100;投诉/维权电话:4009-655-100。