Dtjytyk

Let $Gammasubsetmathrm O(Bbb R^n)$ be a finite group of orthogonal matrices. Let $U_1,U_2subseteqBbb R^n$ be two irreducible invariant subspaces w.r.t. $Gamma$ with $U_1cap U_2={0}$, which are not orthogonal to each other, i.e. there are $u_iin U_i$ with $langle u_1,u_2rangle not=0$.

I was sceptic about the existence of such, but you can find examples here in a previous question of mine. Thinking a bit about such subspaces, I came to the following question:

Question: Is it true, that:

1. 
   $dim U_1=dim U_2=:d$.


2. Every other $d$-dimensional subspace $Usubset U_1oplus U_2$ with $Ucap U_i={0}$ is an irreducible invariant subspace as well.


3. There are two orthogonal $d$-dimensional irreducible invariant subspaces $bar U_1,bar U_2subset U_1oplus U_2$.

Update

The second statement is probably false, but should be substituted by a different one.

If it would be true we could do something like this: choose a subspace $Usubset U_1oplus U_2$ as described in 2., which is still not orthogonal to $U_1$. Now apply 2. again to the pair $U_1,U$ to show that a certain subspace $bar Usubset U_1oplus U$ is invariant. We can probably choose $bar U$ to non-trivially intersect $U_2$, and hence have found an invariant subspace $bar Ucap U_2subset U_2$ in contradiction to the irreducibility of $U_2$.

I still suspect that there are a lot of irreducible invariant subspaces in $U_1oplus U_2$, but it probably can't be all of them.

Let $Tin O(V)$ be an orthogonal matrix where $dim V>0$. Write its characteristic polynomial $P_T$ as
$$P_T=det(XI-T)=prod_{i=1}^k P_i^{m_i},$$
where the $P_i$ are distinct irreducible factors and the $m_i>0$ their multiplicities. Because $P_T$ is orthogonal it is diagonalizable, hence its minimal polynomial is $prod_{i=1}^kP_i$.

For each $i$ define $U_i:=ker P_i$ and let $T_i$ denote the restriction of $T$ to $U_i$. Then for each $i$ the minimal polynomial of $T_i$ is precisely $P_i$.

Proposition: The $U_i$ are pairwise orthogonal $T$-invariant subspaces and $V=bigoplus_{i=1}^k U_i$.

Proof. See proposition 4.7 of the linked reader.

For every $uin V$ let $U_u$ denote the subspace generated by the set ${T^k(u): kgeq0}$, where $T^0:=I$. Then $U_u$ is the smallest $T$-invariant subspace containing $u$. It is clear that

1. If $U$ is a $T$-invariant subspace and $uin U$, then $U_usubset U$.


2. If $U$ is an irreducible $T$-invariant subspace and $uin U$ is non-zero, then $U_u=U$.

Because the $P_i$ are pairwise coprime, if a $T$-invariant subspace $U$ contains some element
$$u=sum_{i=1}^ku_i
qquadtext{ with } u_iin U_i text{ for each }1leq ileq k,$$
then it also contains $u_i$ for each $1leq ileq k$, and hence it contains the $T$-invariant subspace $U_{u_i}subset U_i$. It follows that every irreducible $T$-invariant subspace is a subspace of some $U_i$. Because the $U_i$ are pairwise orthogonal it follows that non-orthogonal irreducible $T$-invariant subspaces are subspaces of the same $U_i$ for some $i$.

So let $U_1$ and $U_2$ be two non-orthogonal irreducible $T$-invariant subspaces of $U$ with $U_1cap U_2=0$. Then without loss of generality the minimal polynomial of $T$ is an irreducible polynomial $P$.

For $uin U$ let $T_u$ denote the restriction of $T$ to $U_u$. Then $P(T_u)=0$ so the minimal polynomial of $T_u$ divides $P$. But $P$ is irreducible, so the minimal polynomial of $T_u$ is also $P$ (unless $u=0$, then it is $1$). This implies that $dim U_u=deg P$, and hence every non-zero irreducible $T$-invariant subspace has dimension $deg P$. In particular $dim U_1=dim U_2=deg P$, proving the first statement.

The second statement holds if $d=1$ but fails if $d>1$:

If $d=1$ then for every $1$-dimensional subspace $Usubset U_1oplus U_2$ we have $U=langle urangle=U_u$ for every non-zero $uin U$. This shows that every $1$-dimensional subspace of $U_1oplus U_2$ is $T$-invariant, and of course it is irreducible.

For $d=2$ this fails; a counterexample for $n=4$ is given by the matrix
$$T:=begin{pmatrix}
0&-1&0&0\
1&hphantom{-}0&0&0\
0&0&0&-1\
0&0&1&hphantom{-}0
end{pmatrix},$$
with the non-orthogonal irreducible $T$-invariant subspaces
$$U_1:=langle e_1,e_2rangle
qquadtext{ and }qquad
U_2:=langle e_1+e_3,e_2+e_4rangle.$$
Here, the $2$-dimensional subspace $langle e_3,e_4ranglesubset U_1oplus U_2$ is not $T$-invariant.

What is true, is that for every $uin U_1oplus U_2$ the subspace $U_u$ is irreducible and $T$-invariant, and moreover that every (non-zero) irreducible $T$-invariant subspace of $U_1oplus U_2$ is of this form.

Note that $d>2$ does not occur over the real numbers, because there are no irreducible polynomials $PinBbb{R}[X]$ with $deg P>2$.

3. There are two orthogonal $d$-dimensional irreducible invariant subspaces $bar U_1,bar U_2subset U_1oplus U_2$.

This is true, and even stronger; there exists a $d$-dimensional $T$-invariant subspace $U_2'subset U_1oplus U_2$ that is orthogonal to $U_1$. For $d=1$ this is easier to see:

If $d=1$ then for $iin{1,2}$ let $u_iin U_i$ with $||u_i||=1$. Then $U_i=langle u_irangle$, and setting
$$u:=u_1-frac{1}{langle u_1,u_2rangle}u_2
qquadtext{ yields }qquad
langle u_1,urangle=0,$$
so $U_u=langle urangle$ is such a subspace.

For $d=2$ I think we can imitate the construction for $d=1$ with some adjustments, but I haven't got a proof (yet).