ih\dZddlZddlmZddlmcmZddlmZddl m Z m Z ddl m Z ddlZddlmZGddZddZdd ZGd d ej*ZGd d ej*ZGddZdZdZdZej8dZddZedk(r edyy)a Entropy-Triggered Adaptive Depth (ETAD) MLP for CIFAR-10 - N-layer MLP with per-layer exit heads - Learns entropy target profile per layer - Skips computation when entropy drops below threshold - Repeat batch iteration with fresh sample injection - Single file: train + test N)datasets transforms) DataLoader) defaultdictceZdZdZdZdZdZdZdZdZ e jjrdnd Z dZd Zd Zd Zd ZdZdZdZdZdZy)Configi  gMbP?cudacpu?333333?g?g?皙?g{Gz?d*N)__name__ __module__ __qualname__ INPUT_DIM HIDDEN_DIMNUM_HIDDEN_LAYERS OUTPUT_DIM BATCH_SIZEEPOCHSLRtorchr is_availableDEVICE N_REPEATSFRESH_RATIO_STARTFRESH_RATIO_ENDENTROPY_TARGET_STARTENTROPY_TARGET_END MIN_DEPTHLAMBDA_ENTROPY LAMBDA_DEPTH LOG_INTERVALSEEDex03.pyrrs{IJJJ F Bzz..0VeFIOINLL Dr.rctj||}tj|tj||zz| S)z3Compute Shannon entropy of probability distributiondim)Fsoftmaxr sumlog)logitsr2epsprobss r/compute_entropyr:9s8 IIf# &E IIeeii 44# > >>r.c8t||jS)zMean entropy across batch)r8)r:mean)r7r8s r/ batch_entropyr=>s 6s + 0 0 22r.c8eZdZfdZddZdZdZdZxZS)ETADMLPc t|||_tj|j |j |_tjt|jDcgc],}tj|j |j .c}|_ tjt|jdzDcgc],}tj|j |j.c}|_ tjtj |j"|j$|jdz|_t)t*|_ycc}wcc}w)N)super__init__confignnLinearrr input_proj ModuleListranger hidden_layersr exit_heads Parameterr linspacer&r'entropy_thresholdsrintlayer_exit_counts)selfrD_ __class__s r/rCzETADMLP.__init__Fs'  ))F$4$4f6G6GH ]]6334,  IIf''):): ;,   --633a78)  IIf''):): ;)  #%,, NN++))((1, # "-S!1-, ) s ,1E41E9c|jd}tj|fdtj|j}d}g}g}t j |j|} |jd| } t| } |j| j|j| j|| |jdk|dk(z} nOd|kr%tj tjn$tj tj} | |dk(z} d|| <|| j}tj | j#d| |}t%|j&D]O\}}|dk(}|j)sn4t j || } |j|dz| } t| } |j| j|j| j|dz}|/||j*j,k\}| |j|k|z} nO||kr%tj| tjn$tj| tj} | |z} ||| <tj | j#d| |}R|dk(}|j*j.||<tj |j#d| |}|j0r:|j3j5D]}|j6|xxdz cc<|r||||fS||fS)a Args: x: Input tensor [B, 3072] return_all: If True, return all layer outputs (for analysis) force_depth: If set, exit at this layer regardless of entropy Returns: logits: Final predictions exit_layers: Which layer each sample exited at entropies: Entropy at each layer (if return_all) r)dtypedeviceNrVrA)sizer fulllongrWr3relurGrKr:appendclonerN zeros_likebool ones_likewhere unsqueeze enumeraterJanyrDr(rtrainingrnumpyrP)rQx return_all force_depth batch_size exit_layers final_logits all_entropies all_logitshr7entropy should_exitexitedllayer active_mask layer_idxcan_exit remainingidxs r/forwardzETADMLP.forwardfsOVVAY jj*%**QXXV     FF4??1% &##A&!&)W]]_-&,,.)  "d&=&=a&@@[TVEVWKMNQ\_%**;ejjIbgbqbqr}FKFPFPcQK r 12 F  !<<>L{{   Q    "$"4"45 HAu%+K??$uQx A+T__QU+A.F%f-G   1   flln -AI"$ (=(==&$*A*A)*LLPXX MVYdMde..wejjIjojyjy{BJOJTJTkU  ;.F"+K !;;  #L3 @ 2% !%!>!> I{{    "    =="(..0 1&&s+q0+ 1 mZG G[((r.cn|jjjjS)z7Return current entropy thresholds (the learned profile))rNdetachrrgrQs r/get_entropy_profilezETADMLP.get_entropy_profiles)&&--/335;;==r.ct|jj}|dk(riSt|jj Dcic] \}}|||z  c}}Scc}}w)z"Return distribution of exit layersr)r5rPvaluessorteditems)rQtotalkvs r/get_exit_distributionzETADMLP.get_exit_distributions[D**1134 A:I)/0F0F0L0L0N)OPA1u9 PPPsA%c8|jjy)z-Reset exit counters for new evaluation periodN)rPclearr~s r/reset_exit_countszETADMLP.reset_exit_countss $$&r.)FN) rrrrCr{rrr __classcell__rSs@r/r?r?Es 2@\)|>Q'r.r?c&eZdZfdZddZxZS)ETADLosscbt|||_tj|_yN)rBrCrDrECrossEntropyLossce)rQrDrSs r/rCzETADLoss.__init__s%  %%'r.c|j||}d}|t|D]\}}||k(} | js|jjt j | dz z|jjz} || | z j} || jz }|t|z}|jj} | |jjz } ||jj|zz|jj| zz}||j||jnd| j| jdfS)z Args: logits: Final predictions [B, 10] targets: Ground truth [B] exit_layers: Which layer each sample exited at [B] entropies: List of entropy tensors per layer (optional) rr) task_loss entropy_loss depth_penalty avg_depth)rrdrerDr&npexpr'absr<lenfloatrr)r*item)rQr7targetsrl entropiesrrrt entropy_l exited_masktarget_entropy entropy_diffrr total_losss r/r{zETADLoss.forwardsnGGFG,   )) 4 8 9*a/ ??$%)[[%E%E$&FFA26N&359[[5S5S&TN%.k$:^$K#P#P#RL L$5$5$77L 8 C N *L %%',,. !DKK$A$AA   KK & & 5 6 KK $ $} 4 5  ")3<3HL--/a*//1")    r.r)rrrrCr{rrs@r/rrs ( * r.rc<eZdZdZ d dZdZdZd dZdZdZ y) RepeatBatchTraineraD Manages repeat batch iteration with fresh sample injection. Each epoch consists of N_REPEAT iterations where: - Early iterations: mostly fresh samples (explore) - Later iterations: mostly hard samples from history (exploit) This prevents memorization while focusing on difficult examples. cn||_||_||_||_||_t j ||_t j |t j|_ t j |t j|_ tjj||_y)NrX) dataset_sizerk n_repeatsfresh_ratio_startfresh_ratio_endr zeros sample_lossesr[ sample_countsr` seen_maskrrandom RandomStaterng)rQrrkrrrseeds r/rCzRepeatBatchTrainer.__init__s($"!2.#[[6"[[UZZH\D99((.r.c|jdkr |jS||jdz z }|jd|z z|j|zzS)z>Linear interpolation from fresh_ratio_start to fresh_ratio_endrA)rrr)rQ iterationts r/get_fresh_ratioz"RepeatBatchTrainer.get_fresh_ratio'sR >>Q )) ) !+ ,%%Q/$2F2F2JJJr.cN|j|}t|j|z}|j|z }|jj dd}t ||krJ|}|t |z }t j|jd|}t j||g}n&t jt |} || d|}d|j|<|dkDr|jj} d| |<t|dz|j} t j| | j} | | } t j| d} | d z}||j!z }|j"j%| j'||j'd }t j(|}n%t j*dt j, }t j||g}t jt |} || S) z Get batch indices mixing hard and fresh samples. Args: iteration: Current repeat iteration (0 to n_repeats-1) Returns: indices: Tensor of sample indices for this batch T)as_tuplerNgr g)minr)rYpreplacerX)rrOrkrnonzerorr randpermrcatrr^rtopkindicesclampr5rchoicerg from_numpyemptyr[)rQr fresh_ration_freshn_hardunseen_indices fresh_indicesn_extra extra_indicespermcandidate_losses n_candidates top_indices hard_lossesweightshard_selection hard_indicesrs r/get_next_indicesz#RepeatBatchTrainer.get_next_indices.s**95 doo 347* >>/22D2A!D ~  (*MM 22G!NN4+<+<=hwGM!II}m&DEM>>#n"56D*4>:M)-}% A:#11779 .2 ] +v{D,=,=>L**%5|DLLK+;7K++ks;K!C'G -G!XX__!!#--/ -N !++N;L ;;q ;L))]L9:~~c'l+t}r.c||zd|z |j|zz|j|<|j|xxdz cc<y)z Update per-sample loss history with EMA. Args: indices: Sample indices that were in the batch losses: Per-sample loss values alpha: EMA update rate (higher = more weight to recent) rAN)rr)rQrlossesalphas r/update_sample_lossesz'RepeatBatchTrainer.update_sample_lossesjsL FNa%i4+=+=g+FF F 7# 7#q(#r.c8|jjy)zReset seen mask for new epochN)rzero_r~s r/ reset_epochzRepeatBatchTrainer.reset_epochxs r.c|jdkD}|jr+|j|jj nd|jr+|j|j j nd|j j |j|dkDj j dS)z6Return statistics about sample difficulty distributionrrA) mean_lossmax_lossn_seenn_seen_multiple)rrerr<rmaxr5)rQseens r/get_difficulty_statsz'RepeatBatchTrainer.get_difficulty_stats|s!!A%CG88:++D1668==?STAE**40446;;=QRhhjoo' $ 2 24 81 <AACHHJ   r.N)r rrr)r) rrr__doc__rCrrrrrr-r.r/rr s/<>BD/$K:x ) r.rctjtjtjddtjdg}tjtjtjddtjdg}t j ddd|}t j ddd|}t||jdd }t||jdd }|||fS) z'Load CIFAR-10 with proper preprocessing)gHPs?gec]?g~jt?)gۊe?ggDio?g|?5^?c$|jdSNrUviewrhs r/z%get_cifar10_loaders..AFF2Jr.c$|jdSrrrs r/rz%get_cifar10_loaders..rr.z../dataT)roottraindownload transformF)rkshuffle num_workers) rComposeToTensor NormalizeLambdarCIFAR10rr)rDtransform_traintransform_test train_dataset test_dataset train_loader test_loaders r/get_cifar10_loadersrs ((53 5./ *O ''53 5./ )N$$ dT_M## ednLm8I8I%)q:L\f6G6G$)q:K m 33r.cx|jtt||j|j|j |j |j|z}|jd}d}d} d} tt} tjtt|D cgc]} || djdc} } tjtt|D cgc] } || d c} }t!d|d|jdt|jD]}|j#|}| |j%|j&}||j%|j&}|j)||d \}}}}t+j,||d }|j/||j1j3|||||\}}|j5|j7||j9z }|j;d }||j=|j?j9z }| |jAdz } | dz } |jCD]\}}| |xx|z cc<|jE|}|jG}|tId|jd zzdk(s||jdz k(st!d|dd|jd|dd|j9ddd|z| z dd|ddd|dd|jK|| z d|z| z d| jCDcic] \}}||| z  c}}d|jGiScc} wcc} wcc}}w)a Train one epoch using repeat batch iteration with fresh sample injection. Each epoch consists of N_REPEATS iterations over batches of size BATCH_SIZE. Early iterations use mostly fresh samples; later iterations focus on hard examples. )rrkrrrrrrUrAu 🔄 Epoch : z repeat iterationsTrinone) reductionr1z Iter 2d/z (fresh=.0%z ) | Loss: .4f | Acc: Y@.2fz% | AvgDepth: rz | MeanDiff: r.3flossaccuracydifficulty_stats)&rrrrr#r$r%r,rrrr stackrIrtensorprintrtor" zero_gradr3 cross_entropyrr}rbackwardsteprargmaxeqr5rYrrrrr)modelr optimizerloss_fnrDepochtrainerrcorrectr total_batchesmetricsiall_data all_targetsrrdatatargetr7rlrrRper_sample_lossesr batch_metricspredrrr diff_statss r/train_epoch_with_repeatsr/s KKM!'$$"" 22.. [[5 G JG EM% G{{%MHZB[\QM!,Q/44R8\]H,,U3}CU=VW a 0 3WXK M%6#3#3"44F GH6++,-> **95 ##FMM2W%((7-2$4,H) YOOFFfM $$W.?.F.F.H.L.L.NO&ffk9Mm  diik! }}}#4776?&&(--// Q !'') DAq AJ!OJ --i8 113 s1f..!34 4 9Y&JZJZ]^J^=^ GIb>6+;+;*<=',-99;s+,w,u,S12,[9#>?)+6s; = >Q->` ]*7NU* -4MMO D&wwv}}-vyy/Gf-2$4,H) Y&ffk9Mm  diik! }}}#4776?&&(--// Q!'') DAq AJ!OJ  v** *a / F5'I;aF }=99;s+6$w,u2DS1IJ,[9#>@ A1A: S[(7NU* +2--/ :$!Q1a#f+o  :  ;sG:c 2|jd}d}d}tt}tt}td} |D]\} } | j |j | j |j } } || d\} } }}|| | | |\}}||j z }| jd}||j| jj z }|| jdz }| jjD]}||xxdz cc<tt| || D][\}\}}}|j }| |dxxdz cc<|j |j k(sL| |dxxdz cc<]|jD]\}}||xx|z cc<|j!i}| jD]\}}|ddkDsd |dz|dz ||<!|t#|z d |z|z t%|jDcic] \}}|||z  c}}|d |jDcic]\}}||t#|z c}}Scc}}wcc}}w) zEvaluate modelrc dddS)Nr)r#rr-r-r.r/rzevaluate..6s Q-Gr.TrrAr1rr#r )rrexit_distributionlayer_accuracy)evalrrrOrr"rrrr5rYrrgrdziprrrr)rr1r rDrr#rr%r6accuracy_per_layerr)r*r7rlrrorr,r-rzr&rtrrrwrrr7statss r/evaluater<-s JJLJG E% G#C($%GH fwwv}}-vyy/Gf5:4D5Q2 Y %ffk9Mmdiik! }}}#4776?&&(--// Q??$**, (C c "a ' " (&c+tV&DE >LAy1aI y )' 2a 7 2vvx1668#"9-i8A=8  > "'') DAq AJ!OJ /4 N&,,.I5 >A  $uY'7 7%. HN1 I S[(7NU*7=>O>U>U>W7XYtq!aUlY(   +2--/ :$!Q1a#f+o  :  Z :s J /JTc & tjtjtj j tjtdtdtjtdtjdtdtj|rKtdtjtdtjdd tjdttt\}}}ttj!tj}t#j$|j'tj( }t"j*j-|d d }t/t}tdtdd}t1dtj2dzD]} |rt5||||t| } nt7||||t| } t9|||t} tddtd| ddtdtd| ddd| dddtd| ddd| dddtd| ddd tjtd!|j;j=d"td#| d$td%| d&|r'd'| vr#| d'} td(| d)d*d+| d,d-| d.| d|kDr;| d}tj>|jA|j;| d/d0|jCtddtd1|ddtdtd2td d$} td3| jED]/\}}d4tG|d5zz}td6|dd7|d8zd9d:|1td;tI| d&jED]\}}td6|dd7|dd| d}d|tjz z }td<|d8zd=d>td?|dd@tjdAyB)Cz Main training loop. Args: use_repeats: If True, use repeat batch iteration with fresh sample injection. If False, use standard DataLoader iteration. u🚀 ETAD MLP for CIFAR-10z Device: z Layers: z hidden + exit headsz Min exit depth: u 🔄 Repeat iterations: u 📊 Fresh ratio: r u → )lrr g?) step_sizegammau📈 Training...zP================================================================================rrA zEPOCH rz SUMMARYz Train Loss: rr r rr%z Test Loss: z Avg Depth: rz / zEntropy Profile: rzExit Distribution: r6zLayer Accuracy: r7rzDifficulty Stats: mean_loss=rrz | n_seen=rz | n_repeat=r)rrNr!zetad_mlp_best.ptu&✅ Training Complete! Best Test Acc: u 📊 FINAL ANALYSISzExit Layer Distribution:u█2z Layer rrz5.1fz% z Accuracy by Exit Layer:u ⚡ Efficiency Gain: z.1fz% fewer layers on averagez (Avg depth: z vs max )N)%r manual_seedrr,rrrrr"rr(r#r$r%rr?roptimAdam parametersr lr_schedulerStepLRrrIrr/r3r<rroundsave state_dictrrrOr) use_repeatsrrrrr schedulerr best_accr! train_metrics test_metricsdiff exit_distrupctbaraccr efficiencys r/mainrYes fkk"IINN6;; &( K  '( K0011E FG  0 01 23 +F,<,<+=>? %f&>&>s%C5I_I_`cHdef G0C6/J,L+} FO  v}} -E 5++-&))\\]^_ L5c:#f>V>V=WXY !%";";"="C"CA"F!GHI #L1D$E#FGH  .>!? @AB -> !34D 0k1B30GH N+<=N8O7PR S  #h .#J/H JJ))+&+&?&?&A"  # O'R Bvh- 28C. BC VH !" (O01I $&oo'< sc#(m# r "SWTN"SE:;<  %'\*:;AACD1 s r "SIQ/01[)Ii&":"::;J #JsN3#77P QR OIc?(63K3K2LA NOr.__main__)rN)rU:0yE>)r[)T) rr torch.nnrEtorch.nn.functional functionalr3 torch.optimrF torchvisionrrtorch.utils.datarrgr collectionsrrr:r=Moduler?rrrr/r3no_gradr<rYrr-r.r/res ,'# D? 3L'biiL'b0 ryy0 jy y |4