Dtjytyk

Theorems like the Lovász Local Lemma?

My adviser wants to be the first author

Current sense amp + op-amp buffer + ADC: Measuring down to 0 with single supply

Official degrees of earth’s rotation per day

How is the Swiss post e-voting system supposed to work, and how was it wrong?

Can elves maintain concentration in a trance?

Distribution of Maximum Likelihood Estimator

Do I need life insurance if I can cover my own funeral costs?

Define, (actually define) the "stability" and "energy" of a compound

Where is the 1/8 CR apprentice in Volo's Guide to Monsters?

Why must traveling waves have the same amplitude to form a standing wave?

What is this large pipe coming out of my roof?

Know when to turn notes upside-down(eighth notes, sixteen notes, etc.)

How could a scammer know the apps on my phone / iTunes account?

How could a female member of a species produce eggs unto death?

Latest web browser compatible with Windows 98

I need to drive a 7/16" nut but am unsure how to use the socket I bought for my screwdriver

Is it possible to upcast ritual spells?

Can anyone tell me why this program fails?

Who is our nearest planetary neighbor, on average?

Bash: What does "masking return values" mean?

Science-fiction short story where space navy wanted hospital ships and settlers had guns mounted everywhere

How can I change step-down my variable input voltage? [Microcontroller]

Will a pinhole camera work with instant film?

Existence and uniqueness of least square fits

Is the set of polynomial with coefficients on $mathbb{Q}$ enumerable?sufficient condition for a polynomial to have roots in $[0,1]$Intuition wrong? Variance of two related sums of random variables.Proving $(a_1,b_1)times (a_2,b_2)timescdotstimes(a_n,b_n)$ is open in $mathbb{R}^n$.Is this property equivalent to absolute continuity?Proving that $R^2to 1$ as the degree of a polynomial $kto infty$ for a least squares regression.Show that multivariate function is convexA special restriction of a non-constant continuous functionIs there a curve fitting algorithm that focuses on fitting one part of the curve well rather than trying to make a good general fit?Deriving Bayesian logistic regression

1

$begingroup$

I just noticed that my pythons scipy library gave me another solution to an exponential curve fit than my pocket calculator.

If you plot both solutions both match the data (visually) very well. This lead me to the question of what can be said about existence and uniqueness of least square fits.

So consider a set of finite points $(x_i,y_i), i in {1,dots,n}, n in mathbb{N}$. And a function $fcolon D supset mathbb{R} to mathbb{R}$ which depends on parameters $a_j, j in mathbb{1,dots,k}, k in mathbb{N}$.

Now one wants to find the parameters $(a_j)$ such that $sum_i (f(x_i) -y_i)^2$ is minimal.

What results are there about existence and uniqueness of $f$?

I am interested in particular in the case that $f$ is an elementary function such as

• polynomial function of the form $f(x) = sum_l a_l x^l$
  


• exponential function of the form $f(x) = a_0 cdot a_1^x$
  


• something reciprocial like $f(x) = a_0 + a_1/x^m$
  


• logarithmic function like $f(x) = a_0 + a_1 ln(x)$
  

If in some of those cases the solution doesn't exist or is not unique, I am looking for simple counterexamples. I am also interested in good references (which includes the proofs) about this special topic.

real-analysis statistics regression

share|cite|improve this question

asked Mar 10 at 14:27

JuliaJulia

4521419

$endgroup$

add a comment | 

1

$begingroup$

I just noticed that my pythons scipy library gave me another solution to an exponential curve fit than my pocket calculator.

If you plot both solutions both match the data (visually) very well. This lead me to the question of what can be said about existence and uniqueness of least square fits.

So consider a set of finite points $(x_i,y_i), i in {1,dots,n}, n in mathbb{N}$. And a function $fcolon D supset mathbb{R} to mathbb{R}$ which depends on parameters $a_j, j in mathbb{1,dots,k}, k in mathbb{N}$.

Now one wants to find the parameters $(a_j)$ such that $sum_i (f(x_i) -y_i)^2$ is minimal.

What results are there about existence and uniqueness of $f$?

I am interested in particular in the case that $f$ is an elementary function such as

• polynomial function of the form $f(x) = sum_l a_l x^l$
  


• exponential function of the form $f(x) = a_0 cdot a_1^x$
  


• something reciprocial like $f(x) = a_0 + a_1/x^m$
  


• logarithmic function like $f(x) = a_0 + a_1 ln(x)$
  

If in some of those cases the solution doesn't exist or is not unique, I am looking for simple counterexamples. I am also interested in good references (which includes the proofs) about this special topic.

real-analysis statistics regression

share|cite|improve this question

asked Mar 10 at 14:27

JuliaJulia

4521419

$endgroup$

add a comment | 

$begingroup$

I just noticed that my pythons scipy library gave me another solution to an exponential curve fit than my pocket calculator.

If you plot both solutions both match the data (visually) very well. This lead me to the question of what can be said about existence and uniqueness of least square fits.

So consider a set of finite points $(x_i,y_i), i in {1,dots,n}, n in mathbb{N}$. And a function $fcolon D supset mathbb{R} to mathbb{R}$ which depends on parameters $a_j, j in mathbb{1,dots,k}, k in mathbb{N}$.

Now one wants to find the parameters $(a_j)$ such that $sum_i (f(x_i) -y_i)^2$ is minimal.

What results are there about existence and uniqueness of $f$?

I am interested in particular in the case that $f$ is an elementary function such as

• polynomial function of the form $f(x) = sum_l a_l x^l$
  


• exponential function of the form $f(x) = a_0 cdot a_1^x$
  


• something reciprocial like $f(x) = a_0 + a_1/x^m$
  


• logarithmic function like $f(x) = a_0 + a_1 ln(x)$
  

If in some of those cases the solution doesn't exist or is not unique, I am looking for simple counterexamples. I am also interested in good references (which includes the proofs) about this special topic.

real-analysis statistics regression

share|cite|improve this question

asked Mar 10 at 14:27

JuliaJulia

4521419

$endgroup$

share|cite|improve this question

asked Mar 10 at 14:27

JuliaJulia

4521419

share|cite|improve this question

asked Mar 10 at 14:27

JuliaJulia

4521419

asked Mar 10 at 14:27

JuliaJulia

4521419

asked Mar 10 at 14:27

JuliaJulia

4521419

1 Answer
1

active

oldest

votes

1

$begingroup$

The dependence of $f$ on $x$ is not very important, it's the dependence on the parameters $a_j$ for each fixed $x$ that counts. In particular, in each of your cases except the second $f$ is linear in the $a_j$, and then $sum_i (f(x_i)-y_i)^2$ is convex in the vector $bf a$. Moreover, unless the $f(x_i)$ are linearly dependent, it is strictly convex, which implies that a minimum is unique, and $sum_i (f(x_i)-y_i)^2 to infty$ as $|bf a| to infty$, which implies that the minimum exists.

share|cite|improve this answer

edited Mar 10 at 14:49

answered Mar 10 at 14:42

Robert IsraelRobert Israel

327k23216470

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thanks. What's about the exponential case?
  $endgroup$
  – Julia
  Mar 10 at 16:36
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Just to be clear, I did not claim the minimum is unique in the exponential case. The residuals $(f(x_i) - y_i)^2$ are not in general convex in that case.
  $endgroup$
  – Robert Israel
  yesterday
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  For the exponential case, note that if all $x_i$ are even integers, $(a_0, a_1)$ and $(a_0, -a_1)$ give the same results. You might not like this example because $f(x)$ is not real when $a_1 < 0$ and $x$ is not an integer. But I suspect examples of nonuniqueness can also be found where the $a_1$ values are positive.
  $endgroup$
  – Robert Israel
  yesterday
  
  
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3142437%2fexistence-and-uniqueness-of-least-square-fits%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

1

$begingroup$

The dependence of $f$ on $x$ is not very important, it's the dependence on the parameters $a_j$ for each fixed $x$ that counts. In particular, in each of your cases except the second $f$ is linear in the $a_j$, and then $sum_i (f(x_i)-y_i)^2$ is convex in the vector $bf a$. Moreover, unless the $f(x_i)$ are linearly dependent, it is strictly convex, which implies that a minimum is unique, and $sum_i (f(x_i)-y_i)^2 to infty$ as $|bf a| to infty$, which implies that the minimum exists.

share|cite|improve this answer

edited Mar 10 at 14:49

answered Mar 10 at 14:42

Robert IsraelRobert Israel

327k23216470

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thanks. What's about the exponential case?
  $endgroup$
  – Julia
  Mar 10 at 16:36
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Just to be clear, I did not claim the minimum is unique in the exponential case. The residuals $(f(x_i) - y_i)^2$ are not in general convex in that case.
  $endgroup$
  – Robert Israel
  yesterday
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  For the exponential case, note that if all $x_i$ are even integers, $(a_0, a_1)$ and $(a_0, -a_1)$ give the same results. You might not like this example because $f(x)$ is not real when $a_1 < 0$ and $x$ is not an integer. But I suspect examples of nonuniqueness can also be found where the $a_1$ values are positive.
  $endgroup$
  – Robert Israel
  yesterday
  
  
  

  
  
  
  

add a comment | 

1

$begingroup$

The dependence of $f$ on $x$ is not very important, it's the dependence on the parameters $a_j$ for each fixed $x$ that counts. In particular, in each of your cases except the second $f$ is linear in the $a_j$, and then $sum_i (f(x_i)-y_i)^2$ is convex in the vector $bf a$. Moreover, unless the $f(x_i)$ are linearly dependent, it is strictly convex, which implies that a minimum is unique, and $sum_i (f(x_i)-y_i)^2 to infty$ as $|bf a| to infty$, which implies that the minimum exists.

share|cite|improve this answer

edited Mar 10 at 14:49

answered Mar 10 at 14:42

Robert IsraelRobert Israel

327k23216470

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thanks. What's about the exponential case?
  $endgroup$
  – Julia
  Mar 10 at 16:36
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Just to be clear, I did not claim the minimum is unique in the exponential case. The residuals $(f(x_i) - y_i)^2$ are not in general convex in that case.
  $endgroup$
  – Robert Israel
  yesterday
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  For the exponential case, note that if all $x_i$ are even integers, $(a_0, a_1)$ and $(a_0, -a_1)$ give the same results. You might not like this example because $f(x)$ is not real when $a_1 < 0$ and $x$ is not an integer. But I suspect examples of nonuniqueness can also be found where the $a_1$ values are positive.
  $endgroup$
  – Robert Israel
  yesterday
  
  
  

  
  
  
  

add a comment | 

$begingroup$

The dependence of $f$ on $x$ is not very important, it's the dependence on the parameters $a_j$ for each fixed $x$ that counts. In particular, in each of your cases except the second $f$ is linear in the $a_j$, and then $sum_i (f(x_i)-y_i)^2$ is convex in the vector $bf a$. Moreover, unless the $f(x_i)$ are linearly dependent, it is strictly convex, which implies that a minimum is unique, and $sum_i (f(x_i)-y_i)^2 to infty$ as $|bf a| to infty$, which implies that the minimum exists.

share|cite|improve this answer

edited Mar 10 at 14:49

answered Mar 10 at 14:42

Robert IsraelRobert Israel

327k23216470

$endgroup$

share|cite|improve this answer

edited Mar 10 at 14:49

answered Mar 10 at 14:42

Robert IsraelRobert Israel

327k23216470

answered Mar 10 at 14:42

Robert IsraelRobert Israel

327k23216470

answered Mar 10 at 14:42

Robert IsraelRobert Israel

327k23216470