Dtjytyk

Having the player face themselves after the mid-game

Why aren't there more Gauls like Obelix?

PTIJ: Sport in the Torah

What would be the most expensive material to an intergalactic society?

Create chunks from an array

What are the characteristics of a glide in English?

Finding the minimum value of a function without using Calculus

Is the differential, dp, exact or not?

Why do phishing e-mails use faked e-mail addresses instead of the real one?

Is there a logarithm base for which the logarithm becomes an identity function?

Is this Paypal Github SDK reference really a dangerous site?

Is it possible to clone a polymorphic object without manually adding overridden clone method into each derived class in C++?

Psychology of 'flow'

How to make sure I'm assertive enough in contact with subordinates?

Are small insurances worth it?

I am the person who abides by rules, but breaks the rules. Who am I?

What can I do if someone tampers with my SSH public key?

What is the oldest European royal house?

(Codewars) Linked Lists-Sorted Insert

Use Mercury as quenching liquid for swords?

Is divide-by-zero a security vulnerability?

What should I do when a paper is published similar to my PhD thesis without citation?

The preposition for the verb (avenge) - avenge sb/sth (on OR from) sb

If nine coins are tossed, what is the probability that the number of heads is even?

Variance of $3$-dimensional vectors

The variance of a sum of random vectorsThe symbols for variance and covarianceWhen can I be sure that the state values estimated from the Kalman Filter have approached the actual values? Is it from the state co-variance matrix?How to prove an equality envolving variance and covarianceVariance propertyHow to calculate covariance from variance?Finding the variance of a function with two vectors of independent random variables.How to obtain rotation matrix from covariance matrixFinding the minimal varianceAutocovariance equal to variance

1

$begingroup$

I am currently optimizing some code and thus, I want to replace an inefficient OpenCV function, which calculates a covariance matrix. The thing is, that I only need the trace of this covariance matrix, as such, I only need the variance, if I am not mistaken.

Well, my main problem is, that each element has an x, y and z coordinate and thus, 3 dimensions. So, my data looks like:

[x_1, y_1, z_1], [x_2, y_2, z_2], ... [x_n, y_n, z_n]

How can I calculate the variance of this 3D data in general - and how could I optimize it to run as fast as possible?

Thanks in advance!

3d covariance variance

share|cite|improve this question

edited Mar 28 '16 at 17:44

Michael Hardy

1

asked Mar 28 '16 at 16:52

IcarusIcarus

61

$endgroup$

bumped to the homepage by Community♦ yesterday

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

add a comment | 

1

$begingroup$

I am currently optimizing some code and thus, I want to replace an inefficient OpenCV function, which calculates a covariance matrix. The thing is, that I only need the trace of this covariance matrix, as such, I only need the variance, if I am not mistaken.

Well, my main problem is, that each element has an x, y and z coordinate and thus, 3 dimensions. So, my data looks like:

[x_1, y_1, z_1], [x_2, y_2, z_2], ... [x_n, y_n, z_n]

How can I calculate the variance of this 3D data in general - and how could I optimize it to run as fast as possible?

Thanks in advance!

3d covariance variance

share|cite|improve this question

edited Mar 28 '16 at 17:44

Michael Hardy

1

asked Mar 28 '16 at 16:52

IcarusIcarus

61

$endgroup$

bumped to the homepage by Community♦ yesterday

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

add a comment | 

$begingroup$

I am currently optimizing some code and thus, I want to replace an inefficient OpenCV function, which calculates a covariance matrix. The thing is, that I only need the trace of this covariance matrix, as such, I only need the variance, if I am not mistaken.

Well, my main problem is, that each element has an x, y and z coordinate and thus, 3 dimensions. So, my data looks like:

[x_1, y_1, z_1], [x_2, y_2, z_2], ... [x_n, y_n, z_n]

How can I calculate the variance of this 3D data in general - and how could I optimize it to run as fast as possible?

Thanks in advance!

3d covariance variance

share|cite|improve this question

edited Mar 28 '16 at 17:44

Michael Hardy

1

asked Mar 28 '16 at 16:52

IcarusIcarus

61

$endgroup$

share|cite|improve this question

edited Mar 28 '16 at 17:44

Michael Hardy

1

asked Mar 28 '16 at 16:52

IcarusIcarus

61

share|cite|improve this question

edited Mar 28 '16 at 17:44

Michael Hardy

1

asked Mar 28 '16 at 16:52

IcarusIcarus

61

edited Mar 28 '16 at 17:44

Michael Hardy

1

edited Mar 28 '16 at 17:44

Michael Hardy

1

asked Mar 28 '16 at 16:52

IcarusIcarus

61

asked Mar 28 '16 at 16:52

IcarusIcarus

61

1 Answer
1

active

oldest

votes

0

$begingroup$

Generally the variance of a random vector $Xinmathbb R^m$ is
$$
Sigma = operatorname{var}(X) = operatorname{E}( (X-mu)(X-mu)^T) inmathbb R^{mtimes m}quadtext{where } mu=operatorname{E}(X)inmathbb R^m.
$$
Thus $Sigma$ is a symmetric non-negative-definite matrix. Feller's book calls this the variance; some others call it the "covariance matrix" since its entries are the covariances between components of $X$. Some of its properties are similar to those of variances of scalar-valued random variables; for example if $Ainmathbb R^{elltimes m}$ so that $AX$ is an $elltimes 1$ random column vector, then $operatorname{var}(AX) = ASigma A^Tinmathbb R^{elltimesell}$.

You're talking about estimating $Sigma$ based on a sample of size $n$, with $m=3$. Regarding your data points as column vectors $begin{bmatrix} x_k \ y_k \ z_k end{bmatrix}$, for $k=1,ldots,n$, we can let $begin{bmatrix} bar x \ bar y \ bar z end{bmatrix}$ and then look at the matrix
$$
M = begin{bmatrix} ldots, & x_k - bar x, & ldots \ ldots, & y_k - bar y, & ldots \ ldots, & z_k - bar z, & ldots end{bmatrix} in mathbb R^{3times n}.
$$
Then
$$
S = frac 1 n M M^T in mathbb R^{3times 3}
$$
is the maximum-likelihood estimator of $Sigma$ if the $3times 1$ column vectors are from a $3$-dimensional normal distribution. The matrix
$$
tilde S = frac 1 {n-1} M M^T in mathbb R^{3times 3}
$$
is an unbiased estimator of $Sigma$ under far weaker assumptions.

share|cite|improve this answer

answered Mar 28 '16 at 17:09

Michael HardyMichael Hardy

1

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thank you. I am not 100% sure if I understand everything correctly. I need the trace of the covariance matrix, the covariance matrix should be symmetric, which M is not. I expected, that M should have a size of NxNx3. Do I have to use the trace of S or do I have to do even something else?
  $endgroup$
  – Icarus
  Mar 28 '16 at 17:38
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  The matrix $MM^T$ is symmetric and is $3times 3$ in your example. The trace of $S$ would probably serve your purpose, as might also the trace of $displaystyle tilde S$. $qquad$
  $endgroup$
  – Michael Hardy
  Mar 28 '16 at 17:40
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I have implemented a small test environment and used the OpenCV implementation for comparison. I found out, that this implementation gives $MM^T$ from which I used the trace directly as result, and not $S$. Since $S$ is basically just scaling down $MM^T$ this will definitely lead to an error if the size of the vector is varying and these results are compared (both is true in my case). Since I am currently replacing the slow OpenCV implementation by my own, this means, that I really had a bug there, if I am not mistaken. So, in short, I definitely have to use $S$ and not $MM^T$?
  $endgroup$
  – Icarus
  Mar 29 '16 at 10:08
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I have no idea what you mean when you say "the size of the vector is varying".
  $endgroup$
  – Michael Hardy
  Mar 29 '16 at 15:41
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  With the size of my vector I mean $n$ from your formulas. I have an application, which separates a huge data set into smaller ones. The data is separated multiple times into "random" subsets, by calculating and comparing the trace of the covariance matrix of these subsets, the best separation can be found. Thus, $n$ varies depending on how the subsets are separated (since they are not separated into equally sized subsets). Thus, by using $S$ (which divides by $n$) the resulting trace is uniformly scaled, since it is independent of $n$. Am I right?
  $endgroup$
  – Icarus
  Mar 29 '16 at 15:57
  
  
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f1717385%2fvariance-of-3-dimensional-vectors%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

0

$begingroup$

Generally the variance of a random vector $Xinmathbb R^m$ is
$$
Sigma = operatorname{var}(X) = operatorname{E}( (X-mu)(X-mu)^T) inmathbb R^{mtimes m}quadtext{where } mu=operatorname{E}(X)inmathbb R^m.
$$
Thus $Sigma$ is a symmetric non-negative-definite matrix. Feller's book calls this the variance; some others call it the "covariance matrix" since its entries are the covariances between components of $X$. Some of its properties are similar to those of variances of scalar-valued random variables; for example if $Ainmathbb R^{elltimes m}$ so that $AX$ is an $elltimes 1$ random column vector, then $operatorname{var}(AX) = ASigma A^Tinmathbb R^{elltimesell}$.

You're talking about estimating $Sigma$ based on a sample of size $n$, with $m=3$. Regarding your data points as column vectors $begin{bmatrix} x_k \ y_k \ z_k end{bmatrix}$, for $k=1,ldots,n$, we can let $begin{bmatrix} bar x \ bar y \ bar z end{bmatrix}$ and then look at the matrix
$$
M = begin{bmatrix} ldots, & x_k - bar x, & ldots \ ldots, & y_k - bar y, & ldots \ ldots, & z_k - bar z, & ldots end{bmatrix} in mathbb R^{3times n}.
$$
Then
$$
S = frac 1 n M M^T in mathbb R^{3times 3}
$$
is the maximum-likelihood estimator of $Sigma$ if the $3times 1$ column vectors are from a $3$-dimensional normal distribution. The matrix
$$
tilde S = frac 1 {n-1} M M^T in mathbb R^{3times 3}
$$
is an unbiased estimator of $Sigma$ under far weaker assumptions.

share|cite|improve this answer

answered Mar 28 '16 at 17:09

Michael HardyMichael Hardy

1

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thank you. I am not 100% sure if I understand everything correctly. I need the trace of the covariance matrix, the covariance matrix should be symmetric, which M is not. I expected, that M should have a size of NxNx3. Do I have to use the trace of S or do I have to do even something else?
  $endgroup$
  – Icarus
  Mar 28 '16 at 17:38
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  The matrix $MM^T$ is symmetric and is $3times 3$ in your example. The trace of $S$ would probably serve your purpose, as might also the trace of $displaystyle tilde S$. $qquad$
  $endgroup$
  – Michael Hardy
  Mar 28 '16 at 17:40
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I have implemented a small test environment and used the OpenCV implementation for comparison. I found out, that this implementation gives $MM^T$ from which I used the trace directly as result, and not $S$. Since $S$ is basically just scaling down $MM^T$ this will definitely lead to an error if the size of the vector is varying and these results are compared (both is true in my case). Since I am currently replacing the slow OpenCV implementation by my own, this means, that I really had a bug there, if I am not mistaken. So, in short, I definitely have to use $S$ and not $MM^T$?
  $endgroup$
  – Icarus
  Mar 29 '16 at 10:08
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I have no idea what you mean when you say "the size of the vector is varying".
  $endgroup$
  – Michael Hardy
  Mar 29 '16 at 15:41
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  With the size of my vector I mean $n$ from your formulas. I have an application, which separates a huge data set into smaller ones. The data is separated multiple times into "random" subsets, by calculating and comparing the trace of the covariance matrix of these subsets, the best separation can be found. Thus, $n$ varies depending on how the subsets are separated (since they are not separated into equally sized subsets). Thus, by using $S$ (which divides by $n$) the resulting trace is uniformly scaled, since it is independent of $n$. Am I right?
  $endgroup$
  – Icarus
  Mar 29 '16 at 15:57
  
  
  

  
  
  
  

add a comment | 

0

$begingroup$

Generally the variance of a random vector $Xinmathbb R^m$ is
$$
Sigma = operatorname{var}(X) = operatorname{E}( (X-mu)(X-mu)^T) inmathbb R^{mtimes m}quadtext{where } mu=operatorname{E}(X)inmathbb R^m.
$$
Thus $Sigma$ is a symmetric non-negative-definite matrix. Feller's book calls this the variance; some others call it the "covariance matrix" since its entries are the covariances between components of $X$. Some of its properties are similar to those of variances of scalar-valued random variables; for example if $Ainmathbb R^{elltimes m}$ so that $AX$ is an $elltimes 1$ random column vector, then $operatorname{var}(AX) = ASigma A^Tinmathbb R^{elltimesell}$.

You're talking about estimating $Sigma$ based on a sample of size $n$, with $m=3$. Regarding your data points as column vectors $begin{bmatrix} x_k \ y_k \ z_k end{bmatrix}$, for $k=1,ldots,n$, we can let $begin{bmatrix} bar x \ bar y \ bar z end{bmatrix}$ and then look at the matrix
$$
M = begin{bmatrix} ldots, & x_k - bar x, & ldots \ ldots, & y_k - bar y, & ldots \ ldots, & z_k - bar z, & ldots end{bmatrix} in mathbb R^{3times n}.
$$
Then
$$
S = frac 1 n M M^T in mathbb R^{3times 3}
$$
is the maximum-likelihood estimator of $Sigma$ if the $3times 1$ column vectors are from a $3$-dimensional normal distribution. The matrix
$$
tilde S = frac 1 {n-1} M M^T in mathbb R^{3times 3}
$$
is an unbiased estimator of $Sigma$ under far weaker assumptions.

share|cite|improve this answer

answered Mar 28 '16 at 17:09

Michael HardyMichael Hardy

1

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thank you. I am not 100% sure if I understand everything correctly. I need the trace of the covariance matrix, the covariance matrix should be symmetric, which M is not. I expected, that M should have a size of NxNx3. Do I have to use the trace of S or do I have to do even something else?
  $endgroup$
  – Icarus
  Mar 28 '16 at 17:38
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  The matrix $MM^T$ is symmetric and is $3times 3$ in your example. The trace of $S$ would probably serve your purpose, as might also the trace of $displaystyle tilde S$. $qquad$
  $endgroup$
  – Michael Hardy
  Mar 28 '16 at 17:40
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I have implemented a small test environment and used the OpenCV implementation for comparison. I found out, that this implementation gives $MM^T$ from which I used the trace directly as result, and not $S$. Since $S$ is basically just scaling down $MM^T$ this will definitely lead to an error if the size of the vector is varying and these results are compared (both is true in my case). Since I am currently replacing the slow OpenCV implementation by my own, this means, that I really had a bug there, if I am not mistaken. So, in short, I definitely have to use $S$ and not $MM^T$?
  $endgroup$
  – Icarus
  Mar 29 '16 at 10:08
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I have no idea what you mean when you say "the size of the vector is varying".
  $endgroup$
  – Michael Hardy
  Mar 29 '16 at 15:41
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  With the size of my vector I mean $n$ from your formulas. I have an application, which separates a huge data set into smaller ones. The data is separated multiple times into "random" subsets, by calculating and comparing the trace of the covariance matrix of these subsets, the best separation can be found. Thus, $n$ varies depending on how the subsets are separated (since they are not separated into equally sized subsets). Thus, by using $S$ (which divides by $n$) the resulting trace is uniformly scaled, since it is independent of $n$. Am I right?
  $endgroup$
  – Icarus
  Mar 29 '16 at 15:57
  
  
  

  
  
  
  

add a comment | 

$begingroup$

Generally the variance of a random vector $Xinmathbb R^m$ is
$$
Sigma = operatorname{var}(X) = operatorname{E}( (X-mu)(X-mu)^T) inmathbb R^{mtimes m}quadtext{where } mu=operatorname{E}(X)inmathbb R^m.
$$
Thus $Sigma$ is a symmetric non-negative-definite matrix. Feller's book calls this the variance; some others call it the "covariance matrix" since its entries are the covariances between components of $X$. Some of its properties are similar to those of variances of scalar-valued random variables; for example if $Ainmathbb R^{elltimes m}$ so that $AX$ is an $elltimes 1$ random column vector, then $operatorname{var}(AX) = ASigma A^Tinmathbb R^{elltimesell}$.

You're talking about estimating $Sigma$ based on a sample of size $n$, with $m=3$. Regarding your data points as column vectors $begin{bmatrix} x_k \ y_k \ z_k end{bmatrix}$, for $k=1,ldots,n$, we can let $begin{bmatrix} bar x \ bar y \ bar z end{bmatrix}$ and then look at the matrix
$$
M = begin{bmatrix} ldots, & x_k - bar x, & ldots \ ldots, & y_k - bar y, & ldots \ ldots, & z_k - bar z, & ldots end{bmatrix} in mathbb R^{3times n}.
$$
Then
$$
S = frac 1 n M M^T in mathbb R^{3times 3}
$$
is the maximum-likelihood estimator of $Sigma$ if the $3times 1$ column vectors are from a $3$-dimensional normal distribution. The matrix
$$
tilde S = frac 1 {n-1} M M^T in mathbb R^{3times 3}
$$
is an unbiased estimator of $Sigma$ under far weaker assumptions.

share|cite|improve this answer

answered Mar 28 '16 at 17:09

Michael HardyMichael Hardy

1

$endgroup$

share|cite|improve this answer

answered Mar 28 '16 at 17:09

Michael HardyMichael Hardy

1

answered Mar 28 '16 at 17:09

Michael HardyMichael Hardy

1

answered Mar 28 '16 at 17:09

Michael HardyMichael Hardy

1