Dtjytyk

There is a standard result for linear regression that the regression coefficients are given by

$$mathbf{beta}=(mathbf{X^T X})^{-1}mathbf{X^T y}$$

or

$(mathbf{X^T X})mathbf{beta}=mathbf{X^T y} tag{2}label{eq2}$

Scaling the explanatory variables does not affect the predictions. I have tried to show this algebraically as follows.

The response is related to the explanatory variables via the matrix equation
$mathbf{y}=mathbf{X beta} tag{3}label{eq3}$

$mathbf{X}$ is an $n times (p+1)$ matrix of n observations on p explanatory variables. The first column of $mathbf{X}$ is a column of ones.

Scaling the explanatory variables with a $(p+1) times (p+1) $ diagonal matrix $mathbf{D}$, whose entries are the scaling factors
$
mathbf{X^s} = mathbf{XD} tag{4}label{eq4}$

$mathbf{X^s}$ and $mathbf{beta^s}$ satisfy $eqref{eq2}$:

$$(mathbf{D^TX^T XD})mathbf{beta^s} =mathbf{D^TX^T y}$$

so

$$mathbf{X^T XD}mathbf{beta^s} =mathbf{X^T y}$$

$$Rightarrow mathbf{D beta^s} = (mathbf{X^T X)^{-1}}mathbf{X^T y}=mathbf{beta}$$

$Rightarrow mathbf{beta^s}=mathbf{D}^{-1}mathbf{beta} tag{5}label{eq5}$

This means if an explanatory variable is scaled by $d_i$ then the regression coefficient $beta_i$is scaled by $1/d_i$ and the effect of the scaling cancels out, i.e.
considering predictions based on scaled values, and using $eqref{eq4},eqref{eq5},eqref{eq3}$

$$mathbf{y^s}=mathbf{X^s beta^s} = mathbf{X D D^{-1}beta}=mathbf{X beta}=mathbf{y}$$
as expected.

Now to the question.

For logistic regression without any regularization, it is suggested, by doing regressions with and without scaling the same effect is seen

fit <- glm(vs ~ mpg, data=mtcars,family=binomial)



print(fit)



Coefficients:

(Intercept)          mpg  

    -8.8331       0.4304  

mtcars$mpg <- mtcars$mpg * 10



fit <- glm(vs ~ mpg, data=mtcars,family=binomial)



print(fit)



Coefficients:

(Intercept)          mpg  

   -8.83307      0.04304  

When the variable mpg is scaled up by 10, the corresponding coefficient is scaled down by 10.

1. How could this scaling property be proved (or disproved ) algebraically for logistic regression?

I found a similar question relating to the effect on AUC when regularization is used.

2. Is there any point to scaling explanatory variables in logistic regression, in the absence of regularization?

Thanks.

Here is a heuristic idea:

The likelihood for a logistic regression model is
$$ ell(beta|y) propto prod_ileft(frac{exp(x_i'beta)}{1+exp(x_i'beta)}right)^{y_i}left(frac{1}{1+exp(x_i'beta)}right)^{1-y_i}
$$
and the MLE is the arg max of that likelihood. When you scale a regressor, you also need to accordingly scale the coefficients to achieve the original maximal likelihood.

Christoph has a great answer (+1). Just writing this because I can't comment there.

The crucial point here is that the likelihood only depends on the coefficients $beta$ through the linear term $X beta$. This makes the likelihood unable to distinguish between "$X beta$" and $(XD) (D^{-1}beta)$", causing the invariance you've noticed.

To be specific about this, we need to introduce some notation (which we can do since we're writing an answer!). Let $y_i | x_i stackrel{ind.}{sim} mathrm{bernoulli}left[ mathrm{logit}^{-1} (x_i^T beta) right]$ be independent draws according to the logistic regression model, where $x_i in mathbb{R}^{p+1}$ is the measured covariates. Write the likelihood of the $i^{th}$ observation as $l(y_i, x_i^T beta)$.

To introduce the change of coordinates, write $bar{x}_i = D x_i$, where $D$ is diagonal matrix with all diagonal entries nonzero. By definition of maximum likelihood estimation, we know that maximum likelihood estimators $hat{beta}$ of the data ${y_i | x_i}$ satisfy that $$sum_{i=1}^n l(y_i, x_i^T beta) leq sum_{i=1}^n l(y_i, x_i^T hatbeta) tag{1}$$ for all coefficients $beta in mathbb{R}^p$, and that maximum likelihood estimators for the data ${y_i | bar{x}_i}$ satisfy that $$sum_{i=1}^n l(y_i, bar{x}_i^T alpha) leq sum_{i=1}^n l(y_i, bar{x}_i^T hatalpha) tag{2}$$ for all coefficients $alpha in mathbb{R}^p$.

In your argument, you used a closed form of the maximum likelihood estimator to derive the result. It turns out, though, (as Cristoph suggested above), all you need to do is work with the likelihood. Let $hat{beta}$ be a maximum likelihood estimator of the data ${y_i | x_i}$. Now, writing $beta = D alpha$, we can use equation (1) to show that $$sum_{i=1}^n l(y_i, bar{x}_i^T alpha) = sum_{i=1}^n lleft(y_i, (x_i^T D) (D^{-1} beta)right) leq sum_{i=1}^n l(y_i, x_i^T hatbeta) = sum_{i=1}^n l(y_i, bar{x}_i^T D^{-1} hat{beta}).$$ That is, $D^{-1} hat{beta}$ satisfies equation (2) and is therefore a maximum likelihood estimator with respect to the data ${y_i | bar{x}_i}$. This is the invariance property you noticed.

(For what it's worth, there's a lot of room for generalizing this argument beyond logistic regression: did we need independent observations? did we need the matrix $D$ to be diagonal? did we need a binary response? did we need the use logit? What notation would you change for this argument to work in different scenarios?)