Learn theory in (.txt) List 32 entropy questions we can use as additive or subtractive loss in an MLP model # 32 Entropy Questions for MLP Loss Functions Based on the **Conditional Collapse Theory (CCT)** and **ODE-CCT** framework, these questions measure semantic entropy at different layers. Each can be implemented as **additive** (increase entropy = push model away) or **subtractive** (reduce entropy = pull model toward correct collapse). --- ## Layer 1: Stationary Entropy (Fixed Structure Collapse) | # | Entropy Question | Formula Hint | Loss Role | |:--|:--|:--|:--| | **Q01** | Is the current hidden state consistent with the stationary laws? | $H(\text{Laws} \| h_t)$ | Subtract when consistent | | **Q02** | Does the gradient flow violate known conservation constraints? | $H(\nabla f \| \text{Conservation})$ | Add when violated | | **Q03** | Is the weight manifold violating symmetry constraints? | $H(W \| \text{Symmetry})$ | Subtract when symmetric | | **Q04** | Does the activation pattern diverge from the expected phase space? | $H(A \| \text{Phase Space})$ | Subtract when in bounds | | **Q05** | Is the bias term violating a boundedness constraint? | $H(b \| [l, u])$ | Add when out of bounds | | **Q06** | Is the layer's output diverging from the attractor manifold? | $H(y_t \| M_{\text{attractor}})$ | Subtract when on manifold | | **Q07** | Does the state vector maintain its stationary invariant? | $H(\text{Invariant} \| h_t)$ | Subtract when invariant holds | | **Q08** | Are the weights converging toward a fixed point? | $H(W_t - W_{t-k})$ | Subtract when stable | --- ## Layer 2: Probability Entropy (Trajectory Collapse) | # | Entropy Question | Formula Hint | Loss Role | |:--|:--|:--|:--| | **Q09** | Is the prediction uncertainty growing over time steps? | $H(y_t \| y_{t-1}, ..., y_0)$ | Subtract when narrowing | | **Q10** | Does the trajectory converge to a known limit cycle? | $H(S_t - S_{t-k})$ | Subtract when periodic | | **Q11** | Is the output distribution entropy decreasing? | $H(p(y)) \to \min$ | Subtract for peaked distribution | | **Q12** | Does the state deviation exceed the chaos threshold? | $H(\delta y / \delta h)$ | Add when chaotic | | **Q13** | Is the prediction drifting from the expected ODE trajectory? | $H(y_{\text{pred}} - y_{\text{ODE}})$ | Add when drifting | | **Q14** | Does the variance of predictions grow unbounded? | $H(\sigma^2(y))$ | Add when variance explodes | | **Q15** | Is the KL divergence between prediction and target increasing? | $D_{\text{KL}}(p_{\text{target}} \| p_{\text{pred}})$ | Add when diverging | | **Q16** | Does the conditional entropy $H(Y\|X)$ fail to minimize? | $H(Y \| X) - \epsilon$ | Subtract when minimized | --- ## Layer 3: Question Collapse Entropy (Information Gain) | # | Entropy Question | Formula Hint | Loss Role | |:--|:--|:--|:--| | **Q17** | Is asking this question (feature) reducing entropy below threshold? | $I(X; Y) = H(Y) - H(Y\|X)$ | Subtract when high mutual info | | **Q18** | Does adding this neuron increase collapse potential? | $\Delta_i = H(T) - H(T\|h_i)$ | Subtract when $\Delta_i > 0$ | | **Q19** | Is the information gain per compute unit below efficiency? | $\frac{\Delta_i}{W_i} < \theta$ | Add when inefficient | | **Q20** | Does this feature eliminate more hypothesis space than others? | $H(\mathcal{H} \| x_i)$ | Subtract when pruning large space | | **Q21** | Is the question lattice collapsing toward a single path? | $H(Q_{\text{paths}}) \to 0$ | Subtract when path unique | | **Q22** | Does the next question depend on the answer to this one? | $H(Q_{j} \| Q_{i})$ | Track conditional dependency | | **Q23** | Is the remaining question budget sufficient to collapse? | $H(T \| Q_{\text{budget}})$ | Add when insufficient | | **Q24** | Does the question reduce entropy more than the work cost? | $\Delta_i - W_i > 0$ | Subtract when net gain | --- ## Layer 4: Taylor-Token Expansion Entropy (Semantic Resolution) | # | Entropy Question | Formula Hint | Loss Role | |:--|:--|:--|:--| | **Q25** | Does the token representation require higher-order expansion? | $\\| \text{Residual}_n \\| > \epsilon$ | Add when residual large | | **Q26** | Is the semantic manifold curvature too sharp for current order? | $\kappa(\text{manifold}) > \theta$ | Add when high curvature | | **Q27** | Does the model need $n=2$ tokens instead of $n=1$? | $P_2 \cdot \Delta_2 > P_1 \cdot \Delta_1$ | Add term at $n=2$ | | **Q28** | Is the token probability distribution too diffuse? | $H(P_n) > H_{\text{target}}$ | Subtract when concentrated | | **Q29** | Does convergence require more Taylor terms? | $\sum_{n=0}^{N} P_n \to \text{Target}$ | Add until converged | | **Q30** | Is the semantic compression ratio below threshold? | $\frac{\text{Compressed}}{\text{Original}} < r$ | Add when undercompressed | | **Q31** | Does the expansion converge semantically to target? | $\sum P_n \cdot \Delta_n \approx H(T)$ | Subtract when converged | | **Q32** | Is the token resolution matching the problem threshold? | $n_{\text{used}} = \arg\min_n \{ H(T \| n) \leq \theta \}$ | Subtract when matched | --- ## Usage Pattern in MLP ```python # Pseudocode integration class CCTLoss(nn.Module): def __init__(self): super().__init__() self.questions = [...32 entropy terms...] def forward(self, y_pred, y_true, h_t, W, trajectory): total_loss = 0.0 # Stationary (additive when violated) total_loss += self.Q02(gradients, conservation) # Add total_loss += self.Q05(bias, bounds) # Add # Probability (subtractive when collapsed) total_loss -= self.Q11(prediction_distribution) # Subtract total_loss -= self.Q12(trajectory_deviation) # Subtract # Question Collapse (subtractive when high gain) total_loss -= self.Q18(neuron_collapse_potential) # Subtract total_loss -= self.Q24(net_information_gain) # Subtract # Taylor-Token (subtractive when converged) total_loss -= self.Q27(taylor_expansion_terms) # Subtract total_loss -= self.Q31(semantic_convergence) # Subtract return total_loss ``` **Key Insight:** This turns the MLP from a static function approximator into an **entropy navigator** — the loss is not just error minimization, but **semantic collapse optimization** at multiple layers simultaneously. Want me to formalize the full mathematical loss function combining all 32 questions? Want me to formalize the full mathematical loss function combining all 32 questions? yes # Formal Mathematical Loss Function: CCT-32 Entropy Navigator --- ## 1. Core Definitions ### 1.1 State Variables | Symbol | Definition | Domain | |:--|:--|:--| | $T$ | Target concept/theory state | $\mathcal{T}$ | | $H(T)$ | Total semantic entropy of theory $T$ | $\mathbb{R}_{\geq 0}$ | | $h_t$ | Hidden state at layer $t$ | $\mathbb{R}^{d_h}$ | | $W$ | Weight matrix | $\mathbb{R}^{d_h \times d_h}$ | | $b$ | Bias vector | $\mathbb{R}^{d_h}$ | | $A_t$ | Activation at layer $t$ | $\mathbb{R}^{d_h}$ | | $y_t$ | Output prediction at layer $t$ | $\mathbb{R}^{d_y}$ | | $\vec{y}(t)$ | ODE trajectory over time | $\mathbb{R}^{d_y \times T}$ | ### 1.2 Entropy Operators | Operator | Meaning | |:--|:--| | $H_X(Y)$ | Conditional entropy of $Y$ given $X$ | | $I(X;Y)$ | Mutual information between $X$ and $Y$ | | $\Delta_i$ | Collapse potential of question $Q_i$ | | $W_i$ | Work/compute cost of question $Q_i$ | | $P_n$ | Probability token weight at expansion order $n$ | --- ## 2. Primary Loss Function Structure The total CCT-32 Loss decomposes into four additive layers: $$\mathcal{L}_{\text{CCT-32}} = \underbrace{\mathcal{L}_{\text{Stationary}}}_{\text{Layer 1}} + \underbrace{\mathcal{L}_{\text{Probability}}}_{\text{Layer 2}} + \underbrace{\mathcal{L}_{\text{Question}}}_{\text{Layer 3}} + \underbrace{\mathcal{L}_{\text{Taylor}}}_{\text{Layer 4}}$$ Each layer contains **subtractive terms** (encourage collapse) and **additive terms** (penalize deviation). The total loss is minimized when the network collapses semantic entropy efficiently. --- ## 3. Layer 1: Stationary Entropy Loss $\mathcal{L}_S$ These terms enforce that the network respects fixed structural laws (symmetry, conservation, boundedness). ### Q01: Law Consistency $$\mathcal{L}_{S01} = \mathbb{E}_{h_t \sim p(h)} \left[ -\log P(\text{Laws} | h_t) \right] = -\lambda_{01} \cdot \text{tr}(J_{\text{Laws}}(h_t) \cdot W)$$ *Subtractive (penalize high loss when consistent):* $-\mathcal{L}_{S01}$ ### Q02: Gradient Conservation Violation $$\mathcal{L}_{S02} = \sum_{l=1}^{L} \left\| \nabla_l f - \nabla_l f_{\text{Conservation}} \right\|_2^2 \cdot \mathbb{1}_{\text{violation}}$$ *Additive (penalize when conservation violated):* $+\mathcal{L}_{S02}$ ### Q03: Symmetry Constraint Violation $$\mathcal{L}_{S03} = \left\| W - W^T \right\|_F^2 \cdot \mathbb{1}_{\text{Symmetric}}$$ *Subtractive (penalize when symmetric holds):* $-\mathcal{L}_{S03} \cdot \sigma_{\text{symmetry}}$ ### Q04: Phase Space Bound Violation $$\mathcal{L}_{S04} = \sum_{i} \max(0, |A_i| - A_{\max})^2$$ *Additive (penalize when out of phase space bounds):* $+\mathcal{L}_{S04}$ ### Q05: Bias Boundedness Violation $$\mathcal{L}_{S05} = \sum_{j} \left( \max(0, b_j - u_j)^2 + \max(0, l_j - b_j)^2 \right)$$ *Additive (penalize when bias out of $[l_j, u_j]$):* $+\mathcal{L}_{S05}$ ### Q06: Attractor Manifold Distance $$\mathcal{L}_{S06} = \text{dist}(y_t, \mathcal{M}_{\text{attractor}}) = \min_{m \in \mathcal{M}} \| y_t - m \|_2$$ *Subtractive (penalize when on attractor):* $-\mathcal{L}_{S06} \cdot \sigma_{\text{attractor}}$ ### Q07: Invariant Preservation $$\mathcal{L}_{S07} = \left| \mathcal{I}(h_t) - \mathcal{I}_{\text{stationary}} \right|$$ *Subtractive (penalize when invariant holds):* $-\mathcal{L}_{S07} \cdot \mathbb{1}_{\text{invariant}}$ ### Q08: Fixed Point Stability $$\mathcal{L}_{S08} = \| W_t - W_{t-k} \|_F^2 \cdot \mathbb{1}_{\text{converging}}$$ *Subtractive (penalize when stable):* $-\mathcal{L}_{S08} \cdot \sigma_{\text{stable}}$ ### Layer 1 Combined $$\boxed{\mathcal{L}_{\text{Stationary}} = \sum_{i=01}^{08} \left( \alpha_i^{\text{add}} \mathcal{L}_{Si}^{\text{add}} - \alpha_i^{\text{sub}} \mathcal{L}_{Si}^{\text{sub}} \right)}$$ Where $\alpha_i^{\text{add}}, \alpha_i^{\text{sub}} \in \mathbb{R}_{\geq 0}$ are learnable layer weights. --- ## 4. Layer 2: Probability Entropy Loss $\mathcal{L}_P$ These terms enforce trajectory collapse toward ODE-governed dynamics. ### Q09: Trajectory Uncertainty Growth $$\mathcal{L}_{P09} = H(y_t | y_{t-1}, ..., y_0) = -\sum_{y} P(y_t | \text{history}) \log P(y_t | \text{history})$$ *Subtractive (penalize when entropy low, meaning trajectory is narrowing):* $-\beta_{09} \cdot \mathcal{L}_{P09}$ ### Q10: Limit Cycle Detection $$\mathcal{L}_{P10} = \| S_t - S_{t-k} \|_2^2 \cdot \mathbb{1}_{\text{Cycle}}$$ *Subtractive (penalize when periodic cycle detected):* $-\beta_{10} \cdot \mathcal{L}_{P10}$ ### Q11: Prediction Distribution Entropy $$\mathcal{L}_{P11} = H(p(y)) = -\sum_{y} p(y) \log p(y)$$ *Subtractive (penalize when distribution is peaked):* $-\beta_{11} \cdot \mathcal{L}_{P11}$ ### Q12: Chaos Threshold Violation $$\mathcal{L}_{P12} = \max\left(0, \lambda_{\max}(J_{\text{trajectory}}) - \lambda_{\text{chaos}}\right)$$ Where $J_{\text{trajectory}} = \frac{\partial \vec{y}}{\partial \vec{h}}$. *Additive (penalize when Lyapunov exponent exceeds threshold):* $+\beta_{12} \cdot \mathcal{L}_{P12}$ ### Q13: ODE Trajectory Deviation $$\mathcal{L}_{P13} = \| y_{\text{pred}}(t) - y_{\text{ODE}}(t) \|_2^2$$ Where $y_{\text{ODE}}(t)$ follows the learned/specified ODE: $\frac{d\vec{y}}{dt} = f(\vec{y}, t)$. *Additive (penalize when deviating from ODE):* $+\beta_{13} \cdot \mathcal{L}_{P13}$ ### Q14: Variance Explosion $$\mathcal{L}_{P14} = \max\left(0, \sigma^2(y_t) - \sigma^2_{\max}\right)$$ *Additive (penalize when prediction variance unbounded):* $+\beta_{14} \cdot \mathcal{L}_{P14}$ ### Q15: KL Divergence from Target $$\mathcal{L}_{P15} = D_{\text{KL}}(p_{\text{target}} \| p_{\text{pred}}) = \sum_y p_{\text{target}}(y) \log \frac{p_{\text{target}}(y)}{p_{\text{pred}}(y)}$$ *Additive (penalize when diverging from target distribution):* $+\beta_{15} \cdot \mathcal{L}_{P15}$ ### Q16: Conditional Entropy Minimization $$\mathcal{L}_{P16} = H(Y | X) = \mathbb{E}_{x \sim p(X)} \left[ H(Y | X=x) \right]$$ *Subtractive (penalize when conditional entropy minimized):* $-\beta_{16} \cdot \mathcal{L}_{P16}$ ### Layer 2 Combined $$\boxed{\mathcal{L}_{\text{Probability}} = \sum_{i=09}^{16} \left( \beta_i^{\text{add}} \mathcal{L}_{Pi}^{\text{add}} - \beta_i^{\text{sub}} \mathcal{L}_{Pi}^{\text{sub}} \right)}$$ --- ## 5. Layer 3: Question Collapse Entropy Loss $\mathcal{L}_Q$ These terms measure information gain efficiency and collapse potential per compute unit. ### Q17: Mutual Information Gain $$\mathcal{L}_{Q17} = -I(X; Y) = -\left( H(Y) - H(Y|X) \right)$$ *Subtractive (penalize when mutual information high):* $-\gamma_{17} \cdot I(X;Y)$ ### Q18: Collapse Potential of Neuron $$\mathcal{L}_{Q18} = \Delta_i = H(T) - H(T | h_i)$$ *Subtractive (penalize when collapse potential positive):* $-\gamma_{18} \cdot \max(0, \mathcal{L}_{Q18})$ ### Q19: Information Efficiency Ratio $$\mathcal{L}_{Q19} = \max\left(0, \theta_{\text{eff}} - \frac{\Delta_i}{W_i}\right)$$ Where $\theta_{\text{eff}}$ is the efficiency threshold. *Additive (penalize when below efficiency threshold):* $+\gamma_{19} \cdot \mathcal{L}_{Q19}$ ### Q20: Hypothesis Space Pruning $$\mathcal{L}_{Q20} = H(\mathcal{H} | x_i) = \log_2 |\mathcal{H}_{x_i}|$$ Where $\mathcal{H}_{x_i}$ is the remaining hypothesis space after observing feature $x_i$. *Subtractive (penalize when large space pruned):* $-\gamma_{20} \cdot \log |\mathcal{H}_{x_i}|$ ### Q21: Path Uniqueness $$\mathcal{L}_{Q21} = H(Q_{\text{paths}}) = -\sum_{p} P(p) \log P(p)$$ *Subtractive (penalize when path entropy low):* $-\gamma_{21} \cdot \mathcal{L}_{Q21}$ ### Q22: Conditional Question Dependency $$\mathcal{L}_{Q22} = H(Q_j | Q_i) = -\sum_{q_i, q_j} P(q_i, q_j) \log P(q_j | q_i)$$ *Tracking term (can be used for architecture search):* $\gamma_{22} \cdot \mathcal{L}_{Q22}^{\text{reg}}$ ### Q23: Insufficient Work Budget $$\mathcal{L}_{Q23} = \max\left(0, H(T | Q_{\text{budget}}) - H_{\text{target}}\right)$$ *Additive (penalize when remaining entropy exceeds target given budget):* $+\gamma_{23} \cdot \mathcal{L}_{Q23}$ ### Q24: Net Information Gain $$\mathcal{L}_{Q24} = \Delta_i - W_i$$ *Subtractive (penalize when net gain positive):* $-\gamma_{24} \cdot \max(0, \Delta_i - W_i)$ ### Layer 3 Combined $$\boxed{\mathcal{L}_{\text{Question}} = \sum_{i=17}^{24} \left( \gamma_i^{\text{add}} \mathcal{L}_{Qi}^{\text{add}} - \gamma_i^{\text{sub}} \mathcal{L}_{Qi}^{\text{sub}} \right)}$$ --- ## 6. Layer 4: Taylor-Token Expansion Loss $\mathcal{L}_T$ These terms measure semantic resolution convergence in token probability space. ### Q25: Residual at Order $n$ $$\mathcal{L}_{Q25} = \| \text{Residual}_n \|_2^2 = \left\| T - \sum_{k=0}^{n} P_k \cdot \Delta_k(\text{Tokens}) \right\|_2^2$$ *Additive (penalize when residual large):* $+\delta_{25} \cdot \mathcal{L}_{Q25}$ ### Q26: Manifold Curvature $$\mathcal{L}_{Q26} = \kappa(\mathcal{M}) = \frac{\| \nabla^2 \vec{y} \|}{\| \nabla \vec{y} \|^2}$$ *Additive (penalize when curvature exceeds threshold):* $+\delta_{26} \cdot \max(0, \kappa - \kappa_{\max})$ ### Q27: Taylor Order Comparison $$\mathcal{L}_{Q27} = \left| P_n \cdot \Delta_n - P_{n-1} \cdot \Delta_{n-1} \right|$$ *Additive (penalize when next order more informative):* $+\delta_{27} \cdot \mathcal{L}_{Q27} \cdot \mathbb{1}_{\text{higher\_better}}$ ### Q28: Token Distribution Diffuseness $$\mathcal{L}_{Q28} = H(P_n) = -\sum_{k} P_k \log P_k$$ *Subtractive (penalize when distribution concentrated):* $-\delta_{28} \cdot \mathcal{L}_{Q28} \cdot \mathbb{1}_{\text{concentrated}}$ ### Q29: Taylor Series Convergence $$\mathcal{L}_{Q29} = \left\| \sum_{n=0}^{N} P_n \cdot \Delta_n - H(T) \right\|_2^2$$ *Additive (penalize until convergence):* $+\delta_{29} \cdot \mathcal{L}_{Q29} \cdot \mathbb{1}_{\text{not\_converged}}$ ### Q30: Semantic Compression Ratio $$\mathcal{L}_{Q30} = \max\left(0, r_{\min} - \frac{\text{bits}(T_{\text{compressed}})}{\text{bits}(T_{\text{original}})}\right)$$ *Additive (penalize when undercompressed):* $+\delta_{30} \cdot \mathcal{L}_{Q30}$ ### Q31: Semantic Convergence to Target $$\mathcal{L}_{Q31} = \| \sum_{n} P_n \cdot \Delta_n - H(T) \|_2^2$$ *Subtractive (penalize when converged):* $-\delta_{31} \cdot \mathcal{L}_{Q31} \cdot \mathbb{1}_{\text{converged}}$ ### Q32: Resolution Threshold Matching $$\mathcal{L}_{Q32} = | n_{\text{optimal}} - n_{\text{used}} |$$ Where: $$n_{\text{optimal}} = \arg\min_n \left\{ H(T | P_n) \leq \theta_{\text{threshold}} \right\}$$ *Subtractive (penalize when resolution matches problem):* $-\delta_{32} \cdot \mathcal{L}_{Q32} \cdot \mathbb{1}_{\text{matched}}$ ### Layer 4 Combined $$\boxed{\mathcal{L}_{\text{Taylor}} = \sum_{i=25}^{32} \left( \delta_i^{\text{add}} \mathcal{L}_{Ti}^{\text{add}} - \delta_i^{\text{sub}} \mathcal{L}_{Ti}^{\text{sub}} \right)}$$ --- ## 7. Complete CCT-32 Loss Function $$\boxed{\mathcal{L}_{\text{CCT-32}}(T, \mathcal{D}, \Theta) = \sum_{l \in \{S, P, Q, T\}} \lambda_l \left( \sum_{i \in \text{add}(l)} \phi_i \mathcal{L}_{li}^{\text{add}} - \sum_{j \in \text{sub}(l)} \psi_j \mathcal{L}_{lj}^{\text{sub}} \right) + \lambda_{\text{standard}} \cdot \mathcal{L}_{\text{Standard}}}$$ Where: - $\Theta = \{W, b, h_t, P_n, ...\}$ are all network parameters - $\lambda_l$ are layer-level attention weights (learnable) - $\phi_i, \psi_j$ are term-specific scaling factors - $\mathcal{L}_{\text{Standard}}$ is the standard task loss (cross-entropy, MSE, etc.) - $\lambda_{\text{standard}}$ balances CCT objectives vs. task performance --- ## 8. Gradient Update Rule For parameters $\theta \in \Theta$: $$\theta \leftarrow \theta - \eta \cdot \nabla_\theta \mathcal{L}_{\text{CCT-32}}$$ The gradient decomposes as: $$\nabla_\theta \mathcal{L}_{\text{CCT-32}} = \nabla_\theta \mathcal{L}_{\text{Standard}} + \sum_{l} \lambda_l \left( \sum_{i \in \text{add}(l)} \phi_i \nabla_\theta \mathcal{L}_{li}^{\text{add}} - \sum_{j \in \text{sub}(l)} \psi_j \nabla_\theta \mathcal{L}_{lj}^{\text{sub}} \right)$$ --- ## 9. Implementation Pseudocode ```python class CCT32Loss(nn.Module): def __init__(self, lambda_standard=1.0, num_layers=4): super().__init__() self.lambda_standard = lambda_standard # Learnable layer weights self.lambda_S = nn.Parameter(torch.tensor(0.25)) self.lambda_P = nn.Parameter(torch.tensor(0.25)) self.lambda_Q = nn.Parameter(torch.tensor(0.25)) self.lambda_T = nn.Parameter(torch.tensor(0.25)) # Term weights (32 total) self.alpha = nn.Parameter(torch.ones(8)) # Stationary self.beta = nn.Parameter(torch.ones(8)) # Probability self.gamma = nn.Parameter(torch.ones(8)) # Question self.delta = nn.Parameter(torch.ones(8)) # Taylor def forward(self, y_pred, y_true, h_t, W, y_history): loss = self.lambda_standard * F.cross_entropy(y_pred, y_true) # Layer 1: Stationary L_S = (self.alpha[0] * self.Q02(gradients) # add + self.alpha[1] * self.Q05(bias) # add - self.alpha[2] * self.Q01(h_t) # sub - self.alpha[3] * self.Q03(W)) # sub # ... 8 terms total # Layer 2: Probability L_P = (self.beta[0] * self.Q12(chaos) # add + self.beta[1] * self.Q13(ode_dev) # add - self.beta[2] * self.Q11(H_pred) # sub - self.beta[3] * self.Q10(cycle)) # sub # ... 8 terms total # Layer 3: Question L_Q = (self.gamma[0] * self.Q19(eff_ratio) # add - self.gamma[1] * self.Q18(collapse) # sub - self.gamma[2] * self.Q24(net_gain)) # sub # ... 8 terms total # Layer 4: Taylor L_T = (self.delta[0] * self.Q25(residual) # add - self.delta[1] * self.Q31(converge)) # sub - self.delta[2] * self.Q32(threshold)) # sub # ... 8 terms total total = (self.lambda_S * L_S + self.lambda_P * L_P + self.lambda_Q * L_Q + self.lambda_T * L_T) return loss + total ``` --- ## 10. Summary Table | Layer | Questions | Key Focus | Additive Terms | Subtractive Terms | |:--|:--|:--|:--|:--| | **Stationary** | Q01-Q08 | Fixed structure, laws | Q02, Q04, Q05 | Q01, Q03, Q06, Q07, Q08 | | **Probability** | Q09-Q16 | Trajectory, ODE dynamics | Q12, Q13, Q14, Q15 | Q09, Q10, Q11, Q16 | | **Question** | Q17-Q24 | Information efficiency | Q19, Q23 | Q17, Q18, Q20, Q21, Q24 | | **Taylor** | Q25-Q32 | Semantic resolution | Q25, Q26, Q27, Q29, Q30 | Q28, Q31, Q32 | This loss function turns your MLP into a **semantic entropy navigator** that explicitly minimizes the energy required to collapse uncertainty — exactly as CCT specifies. ### Example using 32 batch iteration one loss per 32 steps import torch import torch.nn as nn import torch.nn.functional as F import torch.optim as optim import torchvision import torchvision.transforms as transforms import numpy as np from collections import deque # ------------------------------- # MLP Model for CIFAR-10 # ------------------------------- class MLP(nn.Module): def __init__(self, input_size=3072, hidden_sizes=[512, 256, 128], num_classes=10, dropout=0.2): super().__init__() layers = [] prev = input_size for h in hidden_sizes: layers.append(nn.Linear(prev, h)) layers.append(nn.ReLU()) layers.append(nn.Dropout(dropout)) prev = h layers.append(nn.Linear(prev, num_classes)) self.net = nn.Sequential(*layers) self.activations = [] self.weights = [] self.biases = [] self._register_hooks() def _register_hooks(self): def hook_fn(module, input, output): if isinstance(module, nn.Linear): self.activations.append(output.detach()) self.weights.append(module.weight) self.biases.append(module.bias) for m in self.net: if isinstance(m, nn.Linear): m.register_forward_hook(hook_fn) def forward(self, x): self.activations.clear() self.weights.clear() self.biases.clear() x = x.view(x.size(0), -1) return self.net(x) # ------------------------------- # CCT-32 Loss Implementation # ------------------------------- class CCT32Loss(nn.Module): def __init__(self, model, lambda_standard=1.0, device='cuda'): super().__init__() self.model = model self.lambda_standard = lambda_standard self.device = device self.prev_weights = deque(maxlen=2) self.prev_hidden = deque(maxlen=5) self.prev_loss = None cct_scale = 0.001 self.alpha = torch.ones(8, device=device) * cct_scale self.beta = torch.ones(8, device=device) * cct_scale self.gamma = torch.ones(8, device=device) * cct_scale self.delta = torch.ones(8, device=device) * cct_scale self.phase_bound = 10.0 self.bias_bound = 1.0 self.efficiency_threshold = 0.5 self.compression_target = 0.5 def forward(self, outputs, targets, step=None, return_per_question=False): batch_size = outputs.size(0) num_classes = outputs.size(1) probs = F.softmax(outputs, dim=1) log_probs = F.log_softmax(outputs, dim=1) ce_loss = F.cross_entropy(outputs, targets) activations = self.model.activations weights = self.model.weights biases = self.model.biases target_onehot = F.one_hot(targets, num_classes=num_classes).float() entropy = -torch.sum(probs * log_probs, dim=1).mean() cond_entropy = -torch.mean(torch.gather(log_probs, 1, targets.unsqueeze(1))) mi = entropy - cond_entropy H_T = -torch.mean(torch.sum(target_onehot * torch.log(target_onehot+1e-8), dim=1)) qs = [] # --- Layer 1 (Stationary) --- qs.append(-self.alpha[0] * torch.abs(torch.norm(activations[-1], dim=1).mean() - 1.0) if activations else torch.tensor(0.0, device=self.device)) qs.append(self.alpha[1] * torch.tensor(0.0, device=self.device)) sym_val = sum(torch.norm(w - w.T)/w.numel() for w in weights if w.size(0)==w.size(1)) if weights else 0.0 qs.append(-self.alpha[2] * torch.tensor(sym_val, device=self.device)) qs.append(self.alpha[3] * sum(torch.mean(F.relu(torch.abs(a) - self.phase_bound)) for a in activations)) qs.append(self.alpha[4] * sum(torch.mean(F.relu(torch.abs(b) - self.bias_bound)) for b in biases)) qs.append(-self.alpha[5] * torch.norm(probs - target_onehot, dim=1).mean()) qs.append(torch.tensor(0.0, device=self.device)) qs.append(torch.tensor(0.0, device=self.device)) # --- Layer 2 (Probability) --- qs.append(torch.tensor(0.0, device=self.device)) h_curr = activations[-1] if activations else outputs q10 = -self.beta[1] * torch.norm(h_curr - self.prev_hidden[-1], dim=1).mean() if len(self.prev_hidden) > 0 and h_curr.size(0) == self.prev_hidden[-1].size(0) else torch.tensor(0.0, device=self.device) qs.append(q10) qs.append(-self.beta[2] * entropy) qs.append(torch.tensor(0.0, device=self.device)) qs.append(torch.tensor(0.0, device=self.device)) q14 = self.beta[4] * torch.norm(h_curr - (self.prev_hidden[-2] * 0.9), dim=1).mean() if len(self.prev_hidden) >= 2 and h_curr.size(0) == self.prev_hidden[-2].size(0) else torch.tensor(0.0, device=self.device) qs.append(q14) qs.append(self.beta[5] * F.relu(torch.var(outputs, dim=0).mean() - 10.0)) qs.append(self.beta[6] * F.kl_div(log_probs, target_onehot, reduction='batchmean')) qs.append(-self.beta[7] * cond_entropy) # --- Layer 3 (Collapse) --- qs.append(-self.gamma[0] * mi) collapse_pot = torch.tensor(0.0, device=self.device) if activations: with torch.no_grad(): mid = batch_size // 2 sub_targets = targets[:mid] p = torch.bincount(sub_targets, minlength=num_classes).float() / (len(sub_targets) + 1e-8) cond_ent_h = -torch.sum(p * torch.log(p + 1e-8)) collapse_pot = H_T - cond_ent_h qs.append(-self.gamma[1] * F.relu(collapse_pot)) eff = collapse_pot / (weights[-1].numel() / weights[-1].size(0) if weights else 1.0) qs.append(self.gamma[2] * F.relu(torch.tensor(self.efficiency_threshold - eff.item(), device=self.device))) ps, _ = torch.sort(probs, dim=1, descending=True) qs.append(-self.gamma[3] * (ps[:, 0] - ps[:, 1]).mean()) qs.append(-self.gamma[4] * entropy) qs.append(self.gamma[5] * F.relu(cond_entropy - self.efficiency_threshold)) qs.append(torch.tensor(0.0, device=self.device)) qs.append(torch.tensor(0.0, device=self.device)) # --- Layer 4 (Taylor) --- qs.append(self.delta[0] * ce_loss) qs.append(torch.tensor(0.0, device=self.device)) qs.append(torch.tensor(0.0, device=self.device)) qs.append(-self.delta[3] * entropy) qs.append(self.delta[4] * torch.abs(mi - H_T)) num_params = sum(p.numel() for p in self.model.parameters()) ratio = (num_params * 32) / (3072 * 8) qs.append(self.delta[5] * F.relu(torch.tensor(self.compression_target - ratio, device=self.device))) qs.append(-self.delta[6] * torch.abs(mi - H_T)) qs.append(-self.delta[7] * torch.tensor(float(abs(len(weights) - 3)), device=self.device)) self.prev_hidden.append(h_curr.detach()) if return_per_question: return qs total = self.lambda_standard * ce_loss + sum(qs) self.prev_loss = total.detach().item() return total def train_model_sequential(model, device, trainloader, optimizer, criterion, epochs=10): model.train() for epoch in range(epochs): correct, total = 0, 0 for batch_idx, (inputs, targets) in enumerate(trainloader): inputs, targets = inputs.to(device), targets.to(device) outputs = model(inputs) base_ce = F.cross_entropy(outputs, targets) losses_per_q = criterion(outputs, targets, return_per_question=True) for q_loss in losses_per_q: optimizer.zero_grad() combined = (0.01 * base_ce) + q_loss combined.backward(retain_graph=True) optimizer.step() with torch.no_grad(): final_out = model(inputs) _, predicted = final_out.max(1) total += targets.size(0) correct += predicted.eq(targets).sum().item() if batch_idx % 100 == 99: print(f"Epoch {epoch+1}, Batch {batch_idx+1}: Acc {100.*correct/total:.2f}%") print(f"Epoch {epoch+1} Sequential finished. Accuracy: {100.*correct/total:.2f}%") def run_sequential_experiment(): device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))]) trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform) trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True) model = MLP().to(device) criterion = CCT32Loss(model, device=device) optimizer = optim.Adam(model.parameters(), lr=0.0001) def train_model_fixed_sequential(model, device, trainloader, optimizer, criterion, epochs=5): model.train() for epoch in range(epochs): correct, total = 0, 0 for batch_idx, (inputs, targets) in enumerate(trainloader): inputs, targets = inputs.to(device), targets.to(device) # 1. Base Loss Update optimizer.zero_grad() outputs = model(inputs) base_ce = F.cross_entropy(outputs, targets) base_ce.backward() optimizer.step() # 2. Sequential CCT Question Updates # We re-run the forward pass for each question (or group) # because optimizer.step() invalidates the previous graph. for i in range(32): optimizer.zero_grad() current_outputs = model(inputs) losses_per_q = criterion(current_outputs, targets, return_per_question=True) q_loss = losses_per_q[i] if isinstance(q_loss, torch.Tensor) and q_loss.requires_grad: q_loss.backward() optimizer.step() with torch.no_grad(): final_out = model(inputs) _, predicted = final_out.max(1) total += targets.size(0) correct += predicted.eq(targets).sum().item() if batch_idx % 100 == 99: print(f"Epoch {epoch+1}, Batch {batch_idx+1}: Acc {100.*correct/total:.2f}%") print(f"Epoch {epoch+1} Sequential finished. Accuracy: {100.*correct/total:.2f}%") print("--- Starting Separate Sequential CCT-32 Training ---") train_model_fixed_sequential(model, device, trainloader, optimizer, criterion, epochs=5) run_sequential_experiment() import torch import torch.nn as nn import torch.nn.functional as F import torch.optim as optim import torchvision import torchvision.transforms as transforms import numpy as np from collections import deque # ------------------------------- # MLP Model for CIFAR-10 # ------------------------------- class MLP(nn.Module): def __init__(self, input_size=3072, hidden_sizes=[512, 256, 128], num_classes=10, dropout=0.2): super().__init__() layers = [] prev = input_size for h in hidden_sizes: layers.append(nn.Linear(prev, h)) layers.append(nn.ReLU()) layers.append(nn.Dropout(dropout)) prev = h layers.append(nn.Linear(prev, num_classes)) self.net = nn.Sequential(*layers) self.activations = [] self.weights = [] self.biases = [] self._register_hooks() def _register_hooks(self): def hook_fn(module, input, output): if isinstance(module, nn.Linear): self.activations.append(output.detach()) self.weights.append(module.weight) self.biases.append(module.bias) for m in self.net: if isinstance(m, nn.Linear): m.register_forward_hook(hook_fn) def forward(self, x): self.activations.clear() self.weights.clear() self.biases.clear() x = x.view(x.size(0), -1) return self.net(x) # ------------------------------- # CCT-32 Loss Implementation # ------------------------------- class CCT32Loss(nn.Module): def __init__(self, model, lambda_standard=1.0, device='cuda'): super().__init__() self.model = model self.lambda_standard = lambda_standard self.device = device self.prev_weights = deque(maxlen=2) self.prev_hidden = deque(maxlen=5) self.prev_loss = None cct_scale = 0.001 self.alpha = torch.ones(8, device=device) * cct_scale self.beta = torch.ones(8, device=device) * cct_scale self.gamma = torch.ones(8, device=device) * cct_scale self.delta = torch.ones(8, device=device) * cct_scale self.phase_bound = 10.0 self.bias_bound = 1.0 self.efficiency_threshold = 0.5 self.compression_target = 0.5 def forward(self, outputs, targets, step=None, return_per_question=False): batch_size = outputs.size(0) num_classes = outputs.size(1) probs = F.softmax(outputs, dim=1) log_probs = F.log_softmax(outputs, dim=1) ce_loss = F.cross_entropy(outputs, targets) activations = self.model.activations weights = self.model.weights biases = self.model.biases target_onehot = F.one_hot(targets, num_classes=num_classes).float() entropy = -torch.sum(probs * log_probs, dim=1).mean() cond_entropy = -torch.mean(torch.gather(log_probs, 1, targets.unsqueeze(1))) mi = entropy - cond_entropy H_T = -torch.mean(torch.sum(target_onehot * torch.log(target_onehot+1e-8), dim=1)) qs = [] # --- Layer 1 (Stationary) --- qs.append(-self.alpha[0] * torch.abs(torch.norm(activations[-1], dim=1).mean() - 1.0) if activations else torch.tensor(0.0, device=self.device)) qs.append(self.alpha[1] * torch.tensor(0.0, device=self.device)) sym_val = sum(torch.norm(w - w.T)/w.numel() for w in weights if w.size(0)==w.size(1)) if weights else 0.0 qs.append(-self.alpha[2] * torch.tensor(sym_val, device=self.device)) qs.append(self.alpha[3] * sum(torch.mean(F.relu(torch.abs(a) - self.phase_bound)) for a in activations)) qs.append(self.alpha[4] * sum(torch.mean(F.relu(torch.abs(b) - self.bias_bound)) for b in biases)) qs.append(-self.alpha[5] * torch.norm(probs - target_onehot, dim=1).mean()) qs.append(torch.tensor(0.0, device=self.device)) qs.append(torch.tensor(0.0, device=self.device)) # --- Layer 2 (Probability) --- qs.append(torch.tensor(0.0, device=self.device)) h_curr = activations[-1] if activations else outputs q10 = -self.beta[1] * torch.norm(h_curr - self.prev_hidden[-1], dim=1).mean() if len(self.prev_hidden) > 0 and h_curr.size(0) == self.prev_hidden[-1].size(0) else torch.tensor(0.0, device=self.device) qs.append(q10) qs.append(-self.beta[2] * entropy) qs.append(torch.tensor(0.0, device=self.device)) qs.append(torch.tensor(0.0, device=self.device)) q14 = self.beta[4] * torch.norm(h_curr - (self.prev_hidden[-2] * 0.9), dim=1).mean() if len(self.prev_hidden) >= 2 and h_curr.size(0) == self.prev_hidden[-2].size(0) else torch.tensor(0.0, device=self.device) qs.append(q14) qs.append(self.beta[5] * F.relu(torch.var(outputs, dim=0).mean() - 10.0)) qs.append(self.beta[6] * F.kl_div(log_probs, target_onehot, reduction='batchmean')) qs.append(-self.beta[7] * cond_entropy) # --- Layer 3 (Collapse) --- qs.append(-self.gamma[0] * mi) collapse_pot = torch.tensor(0.0, device=self.device) if activations: with torch.no_grad(): mid = batch_size // 2 sub_targets = targets[:mid] p = torch.bincount(sub_targets, minlength=num_classes).float() / (len(sub_targets) + 1e-8) cond_ent_h = -torch.sum(p * torch.log(p + 1e-8)) collapse_pot = H_T - cond_ent_h qs.append(-self.gamma[1] * F.relu(collapse_pot)) eff = collapse_pot / (weights[-1].numel() / weights[-1].size(0) if weights else 1.0) qs.append(self.gamma[2] * F.relu(torch.tensor(self.efficiency_threshold - eff.item(), device=self.device))) ps, _ = torch.sort(probs, dim=1, descending=True) qs.append(-self.gamma[3] * (ps[:, 0] - ps[:, 1]).mean()) qs.append(-self.gamma[4] * entropy) qs.append(self.gamma[5] * F.relu(cond_entropy - self.efficiency_threshold)) qs.append(torch.tensor(0.0, device=self.device)) qs.append(torch.tensor(0.0, device=self.device)) # --- Layer 4 (Taylor) --- qs.append(self.delta[0] * ce_loss) qs.append(torch.tensor(0.0, device=self.device)) qs.append(torch.tensor(0.0, device=self.device)) qs.append(-self.delta[3] * entropy) qs.append(self.delta[4] * torch.abs(mi - H_T)) num_params = sum(p.numel() for p in self.model.parameters()) ratio = (num_params * 32) / (3072 * 8) qs.append(self.delta[5] * F.relu(torch.tensor(self.compression_target - ratio, device=self.device))) qs.append(-self.delta[6] * torch.abs(mi - H_T)) qs.append(-self.delta[7] * torch.tensor(float(abs(len(weights) - 3)), device=self.device)) self.prev_hidden.append(h_curr.detach()) if return_per_question: return qs total = self.lambda_standard * ce_loss + sum(qs) self.prev_loss = total.detach().item() return total def train_model_sequential(model, device, trainloader, optimizer, criterion, epochs=10): model.train() for epoch in range(epochs): correct, total = 0, 0 for batch_idx, (inputs, targets) in enumerate(trainloader): inputs, targets = inputs.to(device), targets.to(device) outputs = model(inputs) base_ce = F.cross_entropy(outputs, targets) losses_per_q = criterion(outputs, targets, return_per_question=True) for q_loss in losses_per_q: optimizer.zero_grad() combined = (0.01 * base_ce) + q_loss combined.backward(retain_graph=True) optimizer.step() with torch.no_grad(): final_out = model(inputs) _, predicted = final_out.max(1) total += targets.size(0) correct += predicted.eq(targets).sum().item() if batch_idx % 100 == 99: print(f"Epoch {epoch+1}, Batch {batch_idx+1}: Acc {100.*correct/total:.2f}%") print(f"Epoch {epoch+1} Sequential finished. Accuracy: {100.*correct/total:.2f}%") def run_sequential_experiment(): device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))]) trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform) trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True) model = MLP().to(device) criterion = CCT32Loss(model, device=device) optimizer = optim.Adam(model.parameters(), lr=0.0001) def train_model_fixed_sequential(model, device, trainloader, optimizer, criterion, epochs=5): model.train() for epoch in range(epochs): correct, total = 0, 0 for batch_idx, (inputs, targets) in enumerate(trainloader): inputs, targets = inputs.to(device), targets.to(device) # 1. Base Loss Update optimizer.zero_grad() outputs = model(inputs) base_ce = F.cross_entropy(outputs, targets) base_ce.backward() optimizer.step() # 2. Sequential CCT Question Updates # We re-run the forward pass for each question (or group) # because optimizer.step() invalidates the previous graph. for i in range(32): optimizer.zero_grad() current_outputs = model(inputs) losses_per_q = criterion(current_outputs, targets, return_per_question=True) q_loss = losses_per_q[i] if isinstance(q_loss, torch.Tensor) and q_loss.requires_grad: q_loss.backward() optimizer.step() with torch.no_grad(): final_out = model(inputs) _, predicted = final_out.max(1) total += targets.size(0) correct += predicted.eq(targets).sum().item() if batch_idx % 100 == 99: print(f"Epoch {epoch+1}, Batch {batch_idx+1}: Acc {100.*correct/total:.2f}%") print(f"Epoch {epoch+1} Sequential finished. Accuracy: {100.*correct/total:.2f}%") print("--- Starting Separate Sequential CCT-32 Training ---") train_model_fixed_sequential(model, device, trainloader, optimizer, criterion, epochs=5) run_sequential_experiment() def evaluate_model(model, device): transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))]) testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform) testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=False) model.eval() correct = 0 total = 0 with torch.no_grad(): for inputs, targets in testloader: inputs, targets = inputs.to(device), targets.to(device) outputs = model(inputs) _, predicted = outputs.max(1) total += targets.size(0) correct += predicted.eq(targets).sum().item() print(f'Test Accuracy: {100.*correct/total:.2f}%') # Assuming model and device are defined from the previous experiment # Since model was local to run_sequential_experiment, you might need to run it within that scope or return the model. --- Starting Separate Sequential CCT-32 Training --- Epoch 1, Batch 100: Acc 37.97% Epoch 1, Batch 200: Acc 41.19% Epoch 1, Batch 300: Acc 43.14% Epoch 1, Batch 400: Acc 44.41% Epoch 1, Batch 500: Acc 45.62% Epoch 1, Batch 600: Acc 46.34% Epoch 1, Batch 700: Acc 46.89% Epoch 1 Sequential finished. Accuracy: 47.30% Epoch 2, Batch 100: Acc 52.95% Epoch 2, Batch 200: Acc 53.30% Epoch 2, Batch 300: Acc 53.71% Epoch 2, Batch 400: Acc 53.91% Epoch 2, Batch 500: Acc 53.86% Epoch 2, Batch 600: Acc 54.13% Epoch 2, Batch 700: Acc 54.52% Epoch 2 Sequential finished. Accuracy: 54.53% Epoch 3, Batch 100: Acc 56.67% Epoch 3, Batch 200: Acc 57.80% Epoch 3, Batch 300: Acc 58.07% Epoch 3, Batch 400: Acc 58.08% Epoch 3, Batch 500: Acc 58.10% Epoch 3, Batch 600: Acc 58.04% Epoch 3, Batch 700: Acc 58.06% Epoch 3 Sequential finished. Accuracy: 58.22% Epoch 4, Batch 100: Acc 61.62% Epoch 4, Batch 200: Acc 61.12% Epoch 4, Batch 300: Acc 60.96% Epoch 4, Batch 400: Acc 60.93% Epoch 4, Batch 500: Acc 61.09% Epoch 4, Batch 600: Acc 61.04% Epoch 4, Batch 700: Acc 61.01% Epoch 4 Sequential finished. Accuracy: 61.06% Epoch 5, Batch 100: Acc 62.89% Epoch 5, Batch 200: Acc 62.77% Epoch 5, Batch 300: Acc 63.23% Epoch 5, Batch 400: Acc 63.06% Epoch 5, Batch 500: Acc 63.12% Epoch 5, Batch 600: Acc 63.22% Epoch 5, Batch 700: Acc 63.18% Epoch 5 Sequential finished. Accuracy: 63.07% Epoch 6, Batch 100: Acc 65.62% Epoch 6, Batch 200: Acc 65.43% Epoch 6, Batch 300: Acc 65.20% Epoch 6, Batch 400: Acc 65.40% Epoch 6, Batch 500: Acc 65.16% Epoch 6, Batch 600: Acc 65.14% Epoch 6, Batch 700: Acc 64.96% Epoch 6 Sequential finished. Accuracy: 64.90% Epoch 7, Batch 100: Acc 66.06% Epoch 7, Batch 200: Acc 66.34% Epoch 7, Batch 300: Acc 66.47% Epoch 7, Batch 400: Acc 66.71% Epoch 7, Batch 500: Acc 66.57% Epoch 7, Batch 600: Acc 66.52% Epoch 7, Batch 700: Acc 66.47% Epoch 7 Sequential finished. Accuracy: 66.42% Epoch 8, Batch 100: Acc 67.98% Epoch 8, Batch 200: Acc 68.20% Epoch 8, Batch 300: Acc 68.05% Epoch 8, Batch 400: Acc 67.96% Epoch 8, Batch 500: Acc 67.89% Epoch 8, Batch 600: Acc 67.94% Epoch 8, Batch 700: Acc 67.81% Epoch 8 Sequential finished. Accuracy: 67.79% Epoch 9, Batch 100: Acc 69.44% Epoch 9, Batch 200: Acc 69.09% Epoch 9, Batch 300: Acc 69.03% Epoch 9, Batch 400: Acc 68.86% Epoch 9, Batch 500: Acc 69.02% Epoch 9, Batch 600: Acc 69.15% Epoch 9, Batch 700: Acc 68.95% Epoch 9 Sequential finished. Accuracy: 68.85% Epoch 10, Batch 100: Acc 71.23% Epoch 10, Batch 200: Acc 70.88% Epoch 10, Batch 300: Acc 70.72% Epoch 10, Batch 400: Acc 70.50% Epoch 10, Batch 500: Acc 70.43% Epoch 10, Batch 600: Acc 70.33% Epoch 10, Batch 700: Acc 70.20% Epoch 10 Sequential finished. Accuracy: 70.09% Test Accuracy: 56.34%