Skip to main content

The score and Fisher information

Two coins can give the same estimate with very different precision, and the difference is written in how sharply the log-likelihood bends. Meet the score, prove that it averages zero at the truth, and measure information as its variance or as the expected curvature.
S02.10.1L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: S02.3 Maximum likelihood M06.7 Expectation, variance and moments M05.5 Hessians, curvature and second-order Taylor

What you will be able to do
  • Define the score as the slope of the log-likelihood and prove that its expectation is zero at the true parameter.
  • Define Fisher information as the variance of the score and derive the second form, minus the expected second derivative.
  • Compute the information of a coin toss, 1 / (p(1 - p)), and of an exponential waiting time, 1 / lambda squared, both ways.
  • Explain why information adds over independent observations and why a sharper log-likelihood means a more precise estimate.

The question

One coin lands heads 7 times in 10 tosses; another lands heads 28 times in 40. Both give the same maximum likelihood estimate, \({\hat{p} = 0.7}\), yet nobody would trust the two equally. Where in the likelihood is the difference written, and can it be measured by one number before a single toss is made? The same number later gives every fitted model its standard errors and steers the natural gradient of reinforcement learning.

Intuition

Module S02.3 found an estimate at the top of the log-likelihood. This module asks how sure that top is. Draw the two coins’ curves, each measured from its own peak, and the answer is in their shape.

Two vermillion log-likelihood curves over the chance of heads p from 0.3 to 1, each shifted so that its peak sits at height 0, and both peaking at p equals 0.7 where a diamond marks the estimate. The dashed curve, for 7 heads in 10 tosses, is wide and is labelled curvature 47.6. The solid curve, for 28 heads in 40 tosses, is narrow and is labelled curvature 190.5. A grey line at height minus 2 carries two brackets where each curve has fallen by 2: width 0.53 for 10 tosses and width 0.28 for 40 tosses.
S02.10.1-F1

The same estimate from 10 tosses (dashed) and from 40 (solid). Four times the tosses give four times the curvature at the peak, and the band of values within 2 of the top is about half as wide.

The wide curve barely changes as \(p\) moves away from 0.7: the data from 10 tosses hardly prefer 0.7 to 0.55 or 0.85. The narrow curve drops fast, so the 40 tosses rule those values out. How fast a curve bends at its top is its curvature, minus the second derivative. Its expected value is called the Fisher information, and it is the number we were looking for: a sharp peak means a precise estimate.

Build it up

Step 1. The score. The slope of the log-likelihood has a name of its own, the score,

\[ U(\theta) = \ell'(\theta). \]

For \(k\) heads in \(n\) tosses, \({\ell(p) = k \ln p + (n - k)\ln(1 - p)}\), so

\[ U(p) = \frac{k}{p} - \frac{n - k}{1 - p} = \frac{k - np}{p(1 - p)} . \]

The second form puts both fractions over \({p(1 - p)}\). The MLE is where the score is zero: \({k = np}\), so \({\hat{p} = k/n}\).

Step 2. Before the data, the score is random. Fix \({p = 0.7}\) and imagine the 10 tosses not yet made. The count \(k\) is random, so \({U(0.7) = (k - 7)/0.21}\) is a random variable too. Each possible sample draws its own score curve, and each curve cuts the line \({p = 0.7}\) at its own height.

Two panels. Left, titled Score curves U of p: seven falling blue curves over p from 0.42 to 0.95, one for each count of heads from 4 to 10, with the curve for k equals 7 drawn thick, as a blue note says. A black vertical line marks the true p equals 0.7, and a blue dot marks where each curve crosses it; the dots are spread above and below 0. Right, titled At the truth: mean 0: blue bars for the probability of each value of U at 0.7, from minus 33 to plus 14, tallest at 0, with a black line at 0 and an arrow showing plus and minus one standard deviation. A note reads mean 0, variance 47.6, sd 6.90.
S02.10.1-F2

Left: the score curve of each possible sample of 10 tosses, and where it crosses the true \({p = 0.7}\). Right: the exact distribution of those crossing heights. They balance at 0, and their spread, variance 47.6, is the information in 10 tosses.

Step 3. At the true value it averages zero. If the coin really has \({p = 0.7}\), the count has mean \({\E[k] = np}\), so the numerator \({k - np}\) averages zero and so does the score. The curves in the picture cross the truth sometimes above, sometimes below, and on average exactly at zero.

Step 4. Its spread is the information. The variance of the count is \({np(1 - p)}\), and dividing by the constant \({p(1 - p)}\) divides the variance by its square:

\[ \begin{aligned} \operatorname{Var}(U) &= \frac{np(1-p)}{p^2(1-p)^2} \\ &= \frac{n}{p(1 - p)} . \end{aligned} \]

At \({p = 0.7}\) and \({n = 10}\) that is \({10 / 0.21 = 47.6}\).

Step 5. The curvature gives the same number. Differentiate the score once more:

\[ -\ell''(p) = \frac{k}{p^2} + \frac{n - k}{(1 - p)^2}. \]

Its expected value, with \({\E[k] = np}\), is \({n/p + n/(1 - p) = n/(p(1 - p))}\), which is again 47.6. Two different questions, “how much does the score wobble?” and “how sharply does the curve bend on average?”, have one answer. That is not luck, as the formal statement shows.

The formal statement

NoteDefinition: score and Fisher information

Let one observation \(X\) have density (or probability) \({f(x \mid \theta)}\), and write \({\ell_1(\theta) = \ln f(X \mid \theta)}\) for its log-likelihood. The score is \({U_1(\theta) = \ell_1'(\theta)}\). The Fisher information in one observation is

\[ I(\theta) = \operatorname{Var}_\theta\big(U_1(\theta)\big), \]

where the subscript says that \(X\) is drawn with parameter \(\theta\). For \(n\) independent observations the score is \({U(\theta) = \ell'(\theta)}\) and the information is \({I_n(\theta)}\).

The results below need regularity conditions: the set of possible \(x\) must not depend on \(\theta\), \(\theta\) must lie inside its range rather than on an edge, and we must be allowed to differentiate under the integral sign. Lesson S02.10.2 shows what goes wrong without the first.

Result 1: the score has mean zero. Every density integrates to 1, whatever \(\theta\) is:

\[ \int f(x \mid \theta)\, dx = 1 . \]

Differentiate both sides with respect to \(\theta\), taking the derivative inside the integral:

\[ \int \frac{\partial f(x \mid \theta)}{\partial \theta}\, dx = 0 . \]

By the chain rule, the derivative of \({\ln f}\) is the derivative of \(f\) divided by \(f\), so \({\partial f / \partial\theta = f \cdot \partial \ln f / \partial\theta}\). Substituting,

\[ \int \frac{\partial \ln f(x \mid \theta)}{\partial \theta}\, f(x \mid \theta)\, dx = 0 , \]

and the left side is the definition of \({\E_\theta[U_1(\theta)]}\). So the score averages zero at the parameter the data come from. Because its mean is zero, its variance is simply \({I(\theta) = \E_\theta[U_1(\theta)^2]}\).

Result 2: information is the expected curvature. Differentiate the last integral once more. The product rule gives two terms:

\[ \begin{aligned} 0 &= \int \frac{\partial^2 \ln f}{\partial \theta^2}\, f\, dx \\ &\quad + \int \frac{\partial \ln f}{\partial \theta}\,\frac{\partial f}{\partial \theta}\, dx . \end{aligned} \]

In the second term replace \({\partial f / \partial\theta}\) by \({f \cdot \partial \ln f/\partial\theta}\) again. It becomes \({\int (\partial \ln f/\partial\theta)^2 f\,dx = \E_\theta[U_1^2]}\), and the first term is \({\E_\theta[\ell_1'']}\). So \({0 = \E_\theta[\ell_1''] + \E_\theta[U_1^2]}\), that is,

\[ \cmodel{I(\theta)} = \E_\theta\big[U_1(\theta)^2\big] = -\E_\theta\big[\ell_1''(\theta)\big]. \]

Result 3: information adds. For independent observations the log-likelihood is a sum, so the score is a sum of \(n\) independent scores, each with mean zero and variance \({I(\theta)}\). Variances of independent terms add (M06.7):

\[ I_n(\theta) = n\, I(\theta). \]

Ten tosses carry ten times the information of one. In a picture, the curvature of the whole log-likelihood grows in proportion to \(n\).

Two panels. Left, titled One toss of a coin: the black U-shaped curve I of p equals 1 over p times 1 minus p, over p from 0 to 1, rising steeply towards both ends, with vermillion dots at p equals 0.5, where it is 4, and at p equals 0.7, where it is 4.76. Right, titled One waiting time: the black curve I of lambda equals 1 over lambda squared, falling from very high near lambda equals 0 towards 0 at lambda equals 2, with a dot at lambda equals 0.5 where it is 4.
S02.10.1-F5

Information depends on where the parameter is. A coin toss is least informative at \({p = 0.5}\) and most informative near 0 or 1; a waiting time says more about a small rate than a large one.

Several parameters. With a vector of parameters the information becomes a matrix, minus the expected Hessian (M05.5) of \({\ell_1}\). For one normal observation with parameters \(\mu\) and \({v = \sigma^2}\),

\[ \mat{I}(\mu, v) = \begin{pmatrix} 1/v & 0 \\ 0 & 1/(2v^2) \end{pmatrix}. \]

The zeros off the diagonal say that, on average, learning about the mean tells you nothing about the variance, and the other way round.

Go deeper: the normal information matrix, entry by entry

One observation has

\[ \begin{aligned} \ell_1 &= -\tfrac12 \ln(2\pi) - \tfrac12 \ln v \\ &\quad - \frac{(x - \mu)^2}{2v} . \end{aligned} \]

Its second derivatives are

\[ \begin{aligned} \frac{\partial^2 \ell_1}{\partial \mu^2} &= -\frac1v , \\ \frac{\partial^2 \ell_1}{\partial \mu\,\partial v} &= -\frac{x - \mu}{v^2} , \\ \frac{\partial^2 \ell_1}{\partial v^2} &= \frac{1}{2v^2} - \frac{(x - \mu)^2}{v^3} . \end{aligned} \]

Using \({\E[X - \mu] = 0}\) and \({\E[(X - \mu)^2] = v}\), their expectations are \({-1/v}\), 0, and \({1/(2v^2)} - {v/v^3} = {-1/(2v^2)}\).

Minus these expectations are the entries of \(\mat{I}(\mu, v)\).

Worked example

Compute the information in one toss at \({p = 0.7}\) both ways, then in ten tosses.

A table with columns toss x, chance, score U, U squared and minus ell double prime. Row head, 1: chance 0.7, score 1 over 0.7 equals 1.429, U squared 2.041, minus ell double prime 1 over 0.49 equals 2.041. Row tail, 0: chance 0.3, score minus 1 over 0.3 equals minus 3.333, U squared 11.111, minus ell double prime 1 over 0.09 equals 11.111. A yellow row headed average gives 0.7 times 1.429 plus 0.3 times minus 3.333 equals 0 for the score, and 4.762 for both U squared and minus ell double prime. Two lines underneath read: both forms give I of 0.7 equals 1 over 0.7 times 0.3 equals 4.762 for one toss; ten tosses: I sub 10 equals 47.6.
S02.10.1-F3

One toss has two outcomes. Weighting each by its chance gives a score that averages 0 and an information of 4.762, whether it is computed from \({U^2}\) or from \({-\ell''}\).

  1. Log-likelihood of one toss. \({\ell_1(p) = x \ln p + (1 - x)\ln(1 - p)}\), with \({x = 1}\) for a head and \({x = 0}\) for a tail.
  2. Score. \({U_1 = x/p - (1 - x)/(1 - p)}\): \({1/0.7 = 1.429}\) after a head, \({-1/0.3 = -3.333}\) after a tail.
  3. Mean. \({0.7 \times 1.429} + {0.3 \times (-3.333)} = 1 - 1 = 0\), as Result 1 promised.
  4. Variance. \({0.7 \times 1.429^2} + {0.3 \times 3.333^2}\), which is \(1.429 + 3.333 = 4.762\).
  5. Curvature. \(-\ell_1'' = {x/p^2} + {(1 - x)/(1 - p)^2}\): 2.041 after a head and 11.111 after a tail, with average \({0.7 \times 2.041} + {0.3 \times 11.111} = 4.762\) again.
  6. In general. \(I(p) = 1/p + {1/(1 - p)}\), which is \({1/(p(1 - p))}\), and \({1/0.21 = 4.762}\).
  7. Ten tosses. \({I_{10}(0.7) = 10 \times 4.762 = 47.6}\), the variance in the build-up figure.

The waiting times. One exponential waiting time has \({\ell_1(\lambda) = \ln \lambda - \lambda x}\), so \({U_1 = 1/\lambda - x}\). Its mean is \({1/\lambda - \E[X] = 0}\), since an exponential waiting time has mean \({1/\lambda}\), and its variance is \({\operatorname{Var}(X) = 1/\lambda^2}\). The curvature is \({-\ell_1'' = 1/\lambda^2}\) for every \(x\), so the second form needs no averaging at all. At \({\lambda = 0.5}\) one waiting time carries information 4, and the five waiting times of the course (1, 3, 0.5, 2.5 and 3 minutes, invented) carry \({I_5 = 20}\).

Code

From scratch: list every outcome of 10 tosses, weight it by its binomial probability, and average the score, its square and the curvature at \({p = 0.7}\).

import math
import numpy as np

n, p = 10, 0.7
ks = range(n + 1)
prob = [math.comb(n, k) * p**k * (1 - p)**(n - k) for k in ks]
score = [k / p - (n - k) / (1 - p) for k in ks]
curv = [k / p**2 + (n - k) / (1 - p)**2 for k in ks]

mean_U = sum(w * u for w, u in zip(prob, score))
E_U2 = sum(w * u * u for w, u in zip(prob, score))
E_curv = sum(w * c for w, c in zip(prob, curv))
print("E[U] =", round(mean_U, 12), "  E[U^2] =", round(E_U2, 3), "  E[-ell''] =", round(E_curv, 3))
print("formula n / (p(1 - p)) =", round(n / (p * (1 - p)), 3), "  one toss:", round(1 / (p * (1 - p)), 3))
E[U] = -0.0   E[U^2] = 47.619   E[-ell''] = 47.619
formula n / (p(1 - p)) = 47.619   one toss: 4.762

Now the waiting times, by simulation: 20,000 samples of five waiting times with rate 0.5 (seed 10), and the score of each sample at the true rate.

rng = np.random.default_rng(10)
lam, m = 0.5, 5
x = rng.exponential(1 / lam, size=(20_000, m))
U = m / lam - x.sum(axis=1)                       # score of the sample: n / lambda - sum of x
print("mean of the scores:", round(float(U.mean()), 2), "  variance:", round(float(U.var()), 2),
      "  theory n / lambda^2 =", m / lam**2)
mean of the scores: -0.0   variance: 19.96   theory n / lambda^2 = 20.0

The simulated mean is within a few hundredths of 0 and the variance within a few tenths of 20; a different seed moves both slightly. SciPy’s frozen distributions have an expect method that does the weighted average for us.

from scipy import stats

I10 = stats.binom(10, 0.7).expect(lambda k: ((k - 7) / 0.21) ** 2)
I1_exp = stats.expon(scale=2).expect(lambda t: (1 / 0.5 - t) ** 2)
print("coin, ten tosses:", round(I10, 3), "  one waiting time at rate 0.5:", round(I1_exp, 3))
coin, ten tosses: 47.619   one waiting time at rate 0.5: 4.0

Explore

This module’s interactive, curvature and intervals, lives in lesson S02.10.4. Before you go, predict: if the coin shows 70 heads in 100 tosses, by what factor does the curvature at the peak grow compared with 7 in 10?

Pitfalls

Two panels of vermillion log-likelihood curves for 7 heads in 10 tosses, dashed, and 28 heads in 40, solid. Left, marked with a cross and titled Compare the heights: the raw curves, the 10-toss curve on top with its peak at minus 6.11 and the 40-toss curve far below with its peak at minus 24.43; a quotation reads higher top, better estimate. Right, marked with a tick and titled Compare the curvatures: the same curves shifted to share a peak at 0; the 10-toss curve is wide and the 40-toss curve narrow.
S02.10.1-F4

Wrong beside right. The height of a log-likelihood’s top falls as data are added, because every observation adds a negative log-probability. Precision is read from the bend, not the height.

“The higher the peak of the log-likelihood, the better the estimate.”

The height depends on how many observations there are and on constants that do not involve the parameter. With 40 tosses the top is at \(-24.43\), far below the \(-6.11\) of 10 tosses, yet the estimate is twice as precise. Compare curves only after shifting each to its own peak, and read precision from the curvature.

“The score is zero at the true parameter.”

The score is zero at the estimate: that is how the MLE is found. At the true value it is zero only on average over repeated samples. For any one sample it is off, by about one standard deviation, \({\sqrt{47.6} = 6.9}\) for 10 tosses, and that wobble is exactly what the information measures.

“Fisher information is computed from my data set.”

\(I(\theta)\) is an expectation over data that could be drawn with parameter \(\theta\), so it is a property of the model at a parameter value, not of one sample. The data’s own version is the observed information \({J(\hat{\theta}) = -\ell''(\hat{\theta})}\), the curvature of the curve you actually have. For the coin and the waiting times the two agree at the MLE; for many models they do not, and lesson S02.10.3 says which to use.

Check yourself

Recall

1. What is the score, and where is it zero for a given sample?

Show answer

The slope of the log-likelihood, \({U(\theta) = \ell'(\theta)}\). It is zero at the maximum likelihood estimate.

2. What is the mean of the score at the true parameter, and why?

Show answer

Zero. Differentiating \({\int f(x \mid \theta)\,dx = 1}\) with respect to \(\theta\) gives \({\int (\partial \ln f/\partial\theta) f\, dx = 0}\), which is \({\E_\theta[U_1] = 0}\).

3. Give the two forms of the Fisher information of one observation.

Show answer

\({I(\theta) = \operatorname{Var}_\theta(U_1) = \E_\theta[U_1^2]}\) and \({I(\theta) = -\E_\theta[\ell_1'']}\).

4. What is the information in one toss of a coin with chance \(p\), and in 10 tosses at \({p = 0.7}\)?

Show answer

\({I(p) = 1/(p(1 - p))}\), which is 4.762 at 0.7; ten tosses carry \({10 \times 4.762 = 47.6}\).

5. What is the information in one exponential waiting time with rate \(\lambda\)?

Show answer

\({I(\lambda) = 1/\lambda^2}\): 4 at \({\lambda = 0.5}\), and 20 for five waiting times.

Exercises

1. ★ A Poisson count has \(P(X = x) = {\lambda^x e^{-\lambda}/x!}\). Find the score of one count and its information, by both forms.

Show solution

\(\ell_1 = {x \ln\lambda} - \lambda - {\ln x!}\), so \({U_1 = x/\lambda - 1}\). Its mean is \({\E[X]/\lambda - 1 = 0}\), and its variance is \({\operatorname{Var}(X)/\lambda^2}\) \(= {\lambda/\lambda^2} = {1/\lambda}\). The curvature is \({-\ell_1'' = x/\lambda^2}\), with expectation \({\lambda/\lambda^2 = 1/\lambda}\). Both forms give \({I(\lambda) = 1/\lambda}\).

2. ★★ A coin lands heads 70 times in 100 tosses. Find the curvature of the log-likelihood at its peak and compare it with 7 heads in 10. By what factor is the band of \({p}\) within 2 of the top narrower, if the curve is treated as a parabola?

Show solution

At \({\hat{p} = 0.7}\), \(-\ell''(\hat p) = {70/0.49} + {30/0.09}\), which is \(142.86 + 333.33 = 476.2\), ten times the 47.6 of 10 tosses. A parabola \({-\tfrac12 J (p - \hat p)^2}\) falls by 2 at \({\abs{p - \hat p} = \sqrt{4/J}}\), so the band shrinks by \({\sqrt{10} \approx 3.16}\): from about \({\pm 0.29}\) to about \({\pm 0.092}\).

3. ★★★ Describe the waiting times by their mean \({\mu = 1/\lambda}\) instead of the rate, so that \({f(x \mid \mu) = e^{-x/\mu}/\mu}\). Find \({I(\mu)}\) directly, and show that it equals \({I(\lambda)\,(d\lambda/d\mu)^2}\).

Show solution

\({\ell_1 = -\ln\mu - x/\mu}\), so \(U_1 = {-1/\mu} + {x/\mu^2} = {(x - \mu)/\mu^2}\). Its variance is \({\operatorname{Var}(X)/\mu^4} = {\mu^2/\mu^4} = {1/\mu^2}\), so \({I(\mu) = 1/\mu^2}\).

With \({\lambda = 1/\mu}\), \({d\lambda/d\mu = -1/\mu^2}\), and

\[ \begin{aligned} I(\lambda)\Big(\frac{d\lambda}{d\mu}\Big)^2 &= \mu^2 \times \frac{1}{\mu^4} \\ &= \frac{1}{\mu^2}. \end{aligned} \]

The two agree. In general the score in \(\mu\) is the score in \(\lambda\) times \({d\lambda/d\mu}\) (the chain rule), so its variance picks up the square of that factor.

Summary card

Information is the expected curvature of the log-likelihood: a sharper peak, a more precise estimate.

The anchor figure again: two log-likelihood curves for the coin, both peaking at 0.7, the one from 10 tosses wide with curvature 47.6 and the one from 40 tosses narrow with curvature 190.5.
S02.10.1-F1

Same estimate, different information.

  • Score. \({U(\theta) = \ell'(\theta)}\), with \({\E_\theta[U] = 0}\) at the truth.
  • Information. \({I(\theta) = \operatorname{Var}(U_1) = -\E[\ell_1'']}\).
  • Adds up. \({I_n(\theta) = n\,I(\theta)}\).
  • Coin and waiting time. \({I(p) = 1/(p(1-p))}\); \({I(\lambda) = 1/\lambda^2}\).

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • Stanford STATS 200, Autumn 2016, Lecture 14: Consistency and asymptotic normality of the MLE. Theorem 14.1 defines the score and the Fisher information by its two equivalent expressions; Lemma 14.1 proves that the score has mean zero and variance minus the expected second derivative, by differentiating the integral of the density, the proof given here.
  • Stanford STATS 200, Autumn 2016, Lecture 15: Fisher information and the Cramer-Rao bound. Section 15.1 gives the information matrix of several parameters (equation 15.1, as covariances of the score or minus expected second derivatives); Section 15.2 reads the information as the expected curvature of the log-likelihood, a sharp curve meaning much information.
  • Philippe Rigollet, Maximum likelihood estimation slides, MIT OpenCourseWare 18.650, Fall 2016. Slide 22 defines the Fisher information as the covariance of the gradient of the one-observation log-likelihood and as minus its expected Hessian, and for one parameter as \({\operatorname{var}[\ell'(\theta)] = -\E[\ell''(\theta)]}\).

Next. S02.10.2 The Cramér-Rao lower bound turns the information into a floor that no unbiased estimator can go below.

This lesson unlocks

S02.10.2 The Cramér-Rao lower bound

Your progress is saved in this browser only. No account, no tracking.
Back to top