de en es fr nl pl pt sv zh

statistics

What did I read in 2025?

Jennifer Lang

I’ve just published my annual reading list for 2025 here, (a little late, but its still January!) as I’ve done for over a decade. Do check out the whole thing, but here are a few of my particular highlights from this year’s books, which, as always, are a bit of an eclectic mixture. Everything is...

book reviews book review statistics feature

Climate change and Mortality – what can we expect?

Jennifer Lang

In Australia, the number of deaths per annum is likely to increase by around 1% by 2030 up to 7% by the end of this century due to global heating. The extra deaths will mostly be at older age groups, and among those who find it harder to shelter from heat. There are likely to be other climate...

statistics demography

No, you most probably don’t know what p-values and confidence intervals mean (but don’t feel bad about it, you can just do Bayesian instead)

Maximilian Speicher

Tales of Design & User Experience (ToDUX) #16 ⁂ Dear readers & friends, Studies have shown over and over that even professional researchers (including statisticians!) can’t correctly interpret a confidence interval or a p-value most of the time. Then, how could you expect the...

python data science tutorial data analysis statistics a/b testing cro online experimentation p-values conversion rate optimization

Numerical stability showcase: Ranking with SoftMax or Boltzmann factor

msuzen

Preamble Image: Babylonian table for computation (Wikipedia)Probably, one of the most important aspects of computational work in quantitative fields, such as physics and data sciences is stability of numerical computations. It implies given inputs, outputs should not wildly deviates to large...

machine learning statistical learning statistical physics numerical analysis connectionist machine learning statistics

Life expectancy drops in Australia

Jennifer Lang

Life expectancy at birth in Australia dropped slightly (by around 0.1-0.2 years to 83.2 at birth),  in the latest calculation from the ABS. That’s not surprising. Mostly due to Covid-19 deaths, mortality was worse in 2022 and 2023 than in the 2010s. The life expectancy number (despite its name) is...

statistics demography covid19

World Excess mortality

Jennifer Lang

Recently, the Mortality Working Group of the Actuaries Institute (of which I’m a member) published our annual research paper on excess mortality in Australia and the world, covering the period 2020-2023.  It’s a great summary, if you are interested in the topic. For this post, I’ve created a...

mortality statistics actuaries covid19

Maintenance Mode & SOFA Lite

Grant Paton-Simpson

SOFA Statistics is now in stable, maintenance mode. New releases are mainly for bug fixes. If you have been having problems running SOFA on Ubuntu / Debian / Linux check out the new deb and archive downloads at https://www.sofastatistics.com/downloads.php Alongside SOFA Statistics I have...

open source python statistics developers

Differentiating ensembles and sample spaces: Alignment between statistical mechanics and probability theory

msuzen

Preamble Sample space is the primary concept introduced in any probability and statistics books and in papers. However, there needs to be more clarity about what constitutes a sample space in general: there is no explicit distinction between the unique event set and the replica sets. The...

thermodynamics probability deep learning statistical physics statistics quantum mechanics statistical mechanics probabilistic models statistical analysis

Overfitting is about complexity ranking of inductive biases : Algorithmic recipe

msuzen

Preamble    Figure: Moon patterns human brain invents. (Wikipedia)Detecting overfitting is inherently a comparison problem of the complexity of multiple objects, i.e., models or an algorithm capable of making predictions. A model is overfitted (underfitted) if we only compare it to another model....

physics machine learning deep learning overfitting statistics interpretable machine learning sparse recovery statistical mechanics statistical models inductive bias

Bayesian rabbit holes: Decoding conditional probability with non-commutative algebra

msuzen

Preamble    The White Rabbit (Wikipedia)A novice analyst or even experienced (data) scientist would have thought that the bar notation $|$ in representing conditional probability carries some different operational mathematics. Primarily when written in explicit distribution functions $p(x|y)$....

data science statistical physics statistics conditional data mining bayesianism statistical mechanics probabilistic models statistical analysis

A misconception in ergodicity: Identify ergodic regime not ergodic process

msuzen

Preamble     Figure 1: Two observable's approach to ergodicity for Bernoulli Trials. Ergodicity appears in many fields, in physics, chemistry and natural sciences but in economics to machine learning as well. Recall that, ergodicity in physics and mathematical definition diverges significantly due...

economics statistical physics statistics bernoulli trails statistical mechanics stochastic processes ergodic hypothesis

Physics origins of the most important statistical ideas of recent times

msuzen

Figure: Maxwell's handwritings, state diagram (Wikipedia)PreambleThe modern statistics now move into an emerging field called data science that amalgamate many different fields from high performance computing to control engineering. However, the emergent behaviour from researchers in machine...

machine learning data science statistical physics statistics data mining interpretable machine learning freshman physics newtonian physics cooperative phenomenon

Full cross-validation and generating learning curves for time-series models

msuzen

Kindly reposted to KDnuggets by Gregory Piatetsky-ShapiroPreambleTime-series analysis is needed almost in any quantitative field and real-life systems that collects data over time, i.e., temporal datasets. Building predictive models on temporal datasets for future evolution of systems in...

machine learning deep learning learning-curve statistical learning learning theory cross-validation time-series statistics temporal

Shifting Modern Data Science Forward: Dijkstra principle for data science

msuzen

Kindly reposted to KDnuggets by Gregory Piatetsky-Shapiro with the title Data science is not about data -applying Dijkstra principle to data science and enhancements.PreludeDijkstra in Zurich, 1984 (Wikipedia)Edsger Dijkstra was a Dutch theoretical physicist turned computer scientist, and probably...

machine learning data science statistics complexity theory scientific computing computer science dijkstra

Computational Epidemiology and Data Scientists: Don't post analysis on outbreak arbitrarily

msuzen

Summary Many data scientist are trained or experienced in using tools to do statistical modelling, forecasting or machine learning solutions, this doesn't necessarily mean that they should just jump out and do an ad-hoc analysis on the available public data on the covid19 outbreak and draw policy...

ethics data science pandemic data analysis disease dynamics s-shaped data. outbreak epidemiology dynamical systems statistics policy

A simple and interpretable performance measure for a binary classifier

msuzen

Kindly reposted to KDnuggets by Gregory Piatetsky-Shapiro  Preamble The core application of machine learning models is a binary classification task. This appears in polyhedra of areas from medicine for diagnostic tests to credit risk decision making for consumers.  Techniques in building...

performance machine learning data science data analysis deep learning statistical learning supervised learning roc ba logistic regression statistics binary classification balanced accuracy pcc auc

The Univariate Fallacy

Zack M. Davis

(originally published at Less Wrong) There's this statistical phenomenon where it's possible for two multivariate distributions to overlap along any one variable, but be cleanly separable when you look at the entire configuration space at once. This is perhaps easiest to see with an illustrative...

mathematics statistics

Nearly ready to release 1.5.0

Grant Paton-Simpson

Version 1.5.0 is nearly ready to release and not before time ;-). The last release was 1.4.6 in January 2016 but the time since then has not been wasted. Here are some of the changes ready to go: SOFA will be able to display worked examples for the following statistical tests:Mann-Whitney...

ubuntu python statistics developers

The fundamental problem of causal inference: causality resource list Looper

msuzen

Preamble One of the main tenants of practical data science is performing statistical inference on data sets which are assumed to be representations of populations of activity, natural or man-made. A specific case is called causal inference, which probably the core interest of decision makers...

machine learning causality rubin causal model randomised control trial matching do calculus uplift modelling bayesian networks statistics causal inference pearl a-null test imputation

Core principles of sustainable data science, machine learning and AI product development: Research as a core driver

msuzen

Kindly reposto to KDnuggets  by Gregory Piatetsky-Shapiro Preamble  Almost all businesses and industry embraced Machine learning (ML) technologies. Apart from ROI concerns, as it is an expensive endeavour to develop and deploy a service driven by ML techniques, sustainability as in going beyond...

ai machine learning data science statistical learning l1 supervised learning linear algebra overtraining overfitting statistics principles. sustainable development

Understanding overfitting: an inaccurate meme in supervised learning

msuzen

Kindly reposted to KDnuggets by Gregory Piatetsky-Shapiro with the title Understanding overfitting: an inaccurate meme in machine learning Preamble There is a lot of confusion among practitioners regarding the concept of overfitting. It seems like, a kind of an urban legend or a meme, a folklore...

machine learning data science prediction statistical learning supervised learning gnu r cross-validation overtraining r programming overfitting regression statistics r r project regularisation