de en es fr nl pl pt sv zh

deep learning

Compressive algorithmic randomness: Gibbs-randomness proposition for massively energy efficient deep learning

msuzen

Figure: Dual Tomographic CompressionPerformance, Süzen, 2025.PreambleRandomness is elusive and its probably one of the outstanding concepts for human scientific endeavour, along with gravity. Kolmogorov complexity, appears to be so novel in trying to answering "what is randomness?". The idea that...

information deep learning complexity randomness statistical mechanics compressed sensing theoretical physics

Building an Autograd Library for Neural Net in C++ from Scratch: My Journey and Lessons

Aditya Chaturvedi

Introduction Creating an Autograd library and Neural nets in C++ from scratch has been one of the most fun and challenging side projects I’ve undertaken recently. Inspired by Andrej Karpathy’s micrograd in Python, I wanted to explore the same idea in C++. The goal was not to replicate Karpathy’s...

mathematics machine learning deep learning gradient-descent neural networks

Resolution of misconception of overfitting: Differentiating learning curves from Occam curves

msuzen

Preamble Occam (Wikipedia)A misconception that overfitted model can be identified with the  amount of generalisation gap between model's training and test sets over its learning curves is still out there. Even in some prominent online lectures and blog posts, this misconception is now repeated...

machine learning deep learning occam's razor complexity theory interpretable machine learning statistical mechanics

Misconceptions on non-temporal learning: When do machine learning models qualify as prediction systems?

msuzen

Preamble    Babylonian Tablet for square root of 2. (Wikipedia)Prediction implies a mechanics, as in knowing a form of a trajectory over time.  Strictly speaking a predictive system implies knowing a solution to the path, set of variable depending on time, time evolution of the system under...

machine learning causality applied mathematics deep learning supervised learning dynamical systems reinforcement learning interpretable machine learning representation learning

The conditional query fallacy: Applying Bayesian inference from discrete mathematics perspective

msuzen

Preamble    The Tilled Field, Joan Miró (Wikipedia)One of the core concepts in data sciences is conditional probabilities, $p(x|y)$ appear as logical description of many of the tasks, such as formulating regression or as a core concept in Bayesian Inference. However, there is operationally no...

machine learning data science deep learning conditional data mining bayesianism discrete mathematics mathematical beauty mathematical philosophy bayesian inference

Differentiating ensembles and sample spaces: Alignment between statistical mechanics and probability theory

msuzen

Preamble Sample space is the primary concept introduced in any probability and statistics books and in papers. However, there needs to be more clarity about what constitutes a sample space in general: there is no explicit distinction between the unique event set and the replica sets. The...

thermodynamics probability deep learning statistical physics statistics quantum mechanics statistical mechanics probabilistic models statistical analysis

Overfitting is about complexity ranking of inductive biases : Algorithmic recipe

msuzen

Preamble    Figure: Moon patterns human brain invents. (Wikipedia)Detecting overfitting is inherently a comparison problem of the complexity of multiple objects, i.e., models or an algorithm capable of making predictions. A model is overfitted (underfitted) if we only compare it to another model....

physics machine learning deep learning overfitting statistics interpretable machine learning sparse recovery statistical mechanics statistical models inductive bias

Building robust AI systems: Is an artificial intelligent agent just a probabilistic boolean function?

msuzen

Preamble    George Boole (Wikipedia)Agent, AI agent or an intelligent agent is used often to describe algorithms or AI systems that are released by research teams recently. However, the definition of an intelligent agent (IA) is a bit opaque. Naïvely thinking, it is nothing more than a decision...

functional programming causality deep learning reinforcement learning interpretable machine learning connectionist learning robustness bayesian statistics.

Empirical risk minimization is not learning : A mathematical definition of learning and re-understanding of overfitting and Occam's razor in machine learning

msuzen

Simionescu Function (Wikipedia)PreambleThe holy grail of machine learning appears to be the empirical risk minimisation. However, on the contrary to general dogma,  the primary objective of machine learning is not risk minimisation per se but mimicking human or animal learning. Empirical risk...

risk machine learning deep learning supervised learning overfitting occam representation learning optimization empirical risk minimisation

Periodic Spectral Ergodicity Accurately Predicts Deep Learning Generalisation

msuzen

Preamble     Dali (1931), The Persistence of Memory (Wikipedia)One of the new mathematical concepts arise due to understanding of deep learning is called periodic spectral ergodicity (PSE). The cascading PSE (cPSE) propagates over deep learning layers which can also be used as a complexity...

deep learning random matrix theory periodic spectral ergodicity complexity theory spectral ergodicity quantum mechanics random matrices ergodic hypothesis

A New Matrix Mathematics for Deep Learning : Random Matrix Theory of Deep Learning

msuzen

Preamble     Figure: Definition of Randomness (Compagner 1991, Delft University)Development of deep learning systems (DLs)  increased our hopes to develop more autonomous systems. Based on the hierarchal learning of representations, deep learning defies the basic learning theory that beg the...

machine learning deep learning linear algebra random matrix theory complexity theory probabilistic models ergodic hypothesis complex networks multidimensional arrays freeman dyson ising-lenz model

Full cross-validation and generating learning curves for time-series models

msuzen

Kindly reposted to KDnuggets by Gregory Piatetsky-ShapiroPreambleTime-series analysis is needed almost in any quantitative field and real-life systems that collects data over time, i.e., temporal datasets. Building predictive models on temporal datasets for future evolution of systems in...

machine learning deep learning learning-curve statistical learning learning theory cross-validation time-series statistics temporal

Conjugacy and Equivalence for Deep Neural Networks: Architecture compression to selection

msuzen

Preamble A recently shown phenomenon can classify deep learning architectures with only using the knowledge gained by trained weights [suezen20a]. The classification produces a measure of equivalence between two trained neural network and astonishingly captures a family of closely related...

learning machine learning deep learning random matrix theory statistical ensemble conjugate matrix ensembles supervised

Book review: A tutorial introduction to the mathematics of deep learning

msuzen

Preamble Artificial Intelligence Engines:An introduction to the Mathematics of Deep Learning by Dr James V. Stone the book and Github repository. (c) 2019 Sebtel Press Deep learning and associated connectionist approaches are now applied routinely in industry and academic research from image...

machine learning deep learning statistical learning statistical physics linear algebra mathematics of neural networks connectionist machine learning classsification

A simple and interpretable performance measure for a binary classifier

msuzen

Kindly reposted to KDnuggets by Gregory Piatetsky-Shapiro  Preamble The core application of machine learning models is a binary classification task. This appears in polyhedra of areas from medicine for diagnostic tests to credit risk decision making for consumers.  Techniques in building...

performance machine learning data science data analysis deep learning statistical learning supervised learning roc ba logistic regression statistics binary classification balanced accuracy pcc auc

Bringing back Occam's razor to modern connectionist machine learning: A simple complexity measure based on statistical physics Cascading Periodic Spectral Ergodicity (cPSE)

msuzen

Kindly reposted to KDnuggets by Gregory Piatetsky-Shapiro with the title Applying Occam's razor to Deep Learning  Kindly reviewed by Cornelius Weber Preamble: Changing concepts in machine learning due to deep learning Occam's razor or principle of parsimony has been the guiding principle in...

deep learning ergodicity model selection spectral analysis neural architecture search connectionist machine learning occam's razor for deep learning principle of parsimony