# THE LARGEST SAMPLE EIGENVALUE DISTRIBUTION IN THE WISHART ENSEMBLE B

#### Document technical information

Format pdf
Size 423.5 kB
First found Jun 9, 2017

#### Document content analysis

Category Also themed
Language
English
Type
not defined
Concepts
no text concepts found

#### Transcript

```The Annals of Probability
2009, Vol. 37, No. 4, 1273–1328
DOI: 10.1214/08-AOP432
© Institute of Mathematical Statistics, 2009
THE LARGEST SAMPLE EIGENVALUE DISTRIBUTION IN THE
RANK 1 QUATERNIONIC SPIKED MODEL OF
WISHART ENSEMBLE
B Y D ONG WANG
Brandeis University
We solve the largest sample eigenvalue distribution problem in the rank 1
spiked model of the quaternionic Wishart ensemble, which is the first case of
a statistical generalization of the Laguerre symplectic ensemble (LSE) on
the soft edge. We observe a phase change phenomenon similar to that in the
complex case, and prove that the new distribution at the phase change point
is the GOE Tracy–Widom distribution.
1. Introduction. The Wishart ensemble is defined as follows [24]:
Consider M independent observation x1 , . . . , xM of an N -variate normal distribution with mean 0 and covariance matrix . Here the values of the normal
distribution can be real, complex or even quaternion. If the variables are complex
or quaternionic, then the definition of the mean is as usual, and the (co)variance is
defined as
cov(x, y) = E (x − x̄)(y − ȳ)∗ ,
where x̄ (resp. ȳ) is the mean of x (resp. y), and ∗ is the complex or quaternionic conjugation operator. Then is a real symmetric/Hermitian/quaternionic
Hermitian matrix. Without loss of generality, we assume to be a diagonal matrix, with population eigenvalues l = (l1 , . . . , lN ). If we put the above data into
an N × M double array X = (x1 : · · · : xM ), then the positively defined real sym1
XX∗ is the sample matrix
metric/Hermitian/quaternionic hermitian matrix S = M
and its eigenvalues λ = (λ1 , . . . , λN ) are sample eigenvalues. (X∗ is the transpose,
Hermitian transpose or quaternionic Hermitian transpose of X depending on type
of X’s entries.) The probability space of λi ’s is called the Wishart ensemble.
It is a classical result [2] that (in the real category) if M N , λi ’s are good
approximations of li ’s. But if M and N are of the same order of magnitude,
that is, M/N = γ 2 ≥ 1 and M and N are very large, the problem is subtler.
The simplest case with = I , the white Wishart ensemble, is the Laguerre ensemble, well studied in random matrix theory (RMT) under the name LOE, LUE
and LSE—they are abbreviations of Laguerre Orthogonal/Unitary/Symplectic Ensemble, and GOE, GUE and GSE appearing later are abbreviations of Gaussian
AMS 2000 subject classifications. Primary 15A52, 41A60; secondary 60F99.
Key words and phrases. Wishart distribution, quaternionic spiked model.
1273
1274
D. WANG
Orthogonal/Unitary/Symplectic Ensemble—over all the three base fields, respectively.
Naturally, the next question is: If is slightly deviate from I , such that li =
1 + ai , i = 1, . . . , k, and lk+1 = · · · = lN = 1, what is the distribution of the λi ’s?
This is called the spiked model [19] and k is defined as its rank.
If M and N are very large and k and ai ’s are small constants, the density of
λi ’s is the same as that in the white Wishart model, proved in [22] in real and
complex categories. The distribution of the largest sample eigenvalue, however,
may change. For the complex ensemble, Baik, Ben Arous and Péché [4] solved the
problem completely. They show that if max(ai ) is smaller than a threshold, then
the distribution of the largest sample eigenvalue is the same as that in the white
ensemble, which is the GUE Tracy–Widom distribution, but if max(ai ) exceeds the
threshold, that distribution is changed into a Gaussian whose mean and variance
depend on max(ai ). Furthermore, in the case that max(ai ) equals the critical value,
they find a series of new distributions, indexed by the multiplicity of max(ai ).
In the real category, which is practically the most important and mathematically
the most difficult, much less is known. In this paper I solve the distribution of the
largest sample eigenvalue for the rank 1 spiked model in the quaternionic category.
I believe the similarity of LOE and LSE [13] suggests that the solution to the
quaternionic spiked model is an intermediate step toward the solution to the real
one.
1.1. Some known results for the largest sample eigenvalue in white and rank 1
spiked models. In latter part of the paper, we concentrate on the distribution of
the largest sample eigenvalue in the rank 1 spiked model, so denote a to be the
only perturbation parameter.
The result in the complex category is complete. First we recall the result for the
complex white Wishart ensemble.
P ROPOSITION 1. The distribution of the largest sample eigenvalue in the complex white Wishart ensemble satisfies that, max(λ) almost surely approaches [15]
(1 + γ −1 )2 with fluctuation scale M −2/3 , and [11, 18]
γ M 2/3
≤T
M→∞
(1 + γ )4/3
where FGUE is the GUE Tracy–Widom distribution.
lim P max(λ) − (1 + γ −1 )2 ·
= FGUE (T ),
The GUE Tracy–Widom distribution is defined by Fredholm determinant [11,
29]:
FGUE (T ) = det 1 − KAiry (ξ, η)χ(T ,∞) (η) ,
where χ(T ,∞) is the step function:
χ(T ,∞) (η) =
1,
0,
if η ∈ (T , ∞),
otherwise,
1275
QUATERNIONIC WISHART
and KAiry (ξ, η) is the well-known Airy kernel defined by the Airy function Ai(x):
KAiry (ξ, η) =
(1)
∞
0
Ai(ξ + t) Ai(η + t) dt.
The Airy function can be defined in different ways, and here we take an integral
representation suitable for our asymptotic analysis [4]:
−1
Ai(ξ ) =
2πi
(2)
e−ξ z+1/3z dz,
3
∞
where ∞ = 1∞ ∪ 2∞ ∪ 3∞ , which are defined as (see Figure 1)
1∞ = {−teπ i/3 |−∞ < t ≤ −1},
2∞ = e−tπ i |− 13 ≤ t ≤
1
3 ,
3∞ = {te5π i/3 |1 ≤ t < ∞}.
The breakthrough in the complex category is by [4], which is for any finite rank
spiked model. In the rank 1 case, it is:
P ROPOSITION 2.
In the rank 1 complex spiked model:
1. If −1 < a < γ −1 , then the distribution of the largest sample eigenvalue is the
same as that of the complex white Wishart ensemble in Proposition 1.
2. If a = γ −1 , then the limit and the fluctuation scale are the same as those of the
complex white Wishart ensemble, but the distribution function is
(3)
lim P max(λ) − (1 + γ
M→∞
−1 2 γ M 2/3
) ·
≤T
(1 + γ )4/3
F IG . 1.
∞ .
= FGUE1 (T ).
1276
D. WANG
3. If a > γ −1 , then the limit and the fluctuation scale are changed as well as the
distribution function, which is a Gaussian:
√
1
M
lim P max(λ) − (a + 1) 1 + 2
·
≤T
M→∞
γ a
(a + 1) 1 − 1/(γ 2 a 2 )
(4)
T
1
2
√ e−t /2 dt.
=
−∞ 2π
The function FGUE1 occurring in (3) is defined similarly to FGUE [4]:
(5)
FGUE1 (T ) = det 1 − KAiry (ξ, η) + s (1) (ξ ) Ai(η) χ(T ,∞) (η) ,
where s (1) is one of a series of functions defined in [4], and has the integral representation
∞
1
(1)
−ηz+1/3z3 1
(1)
s (η) =
dz and s (η) = 1 −
e
Ai(t) dt,
2πi ¯¯ ∞
z
η
where ¯¯ ∞ = ¯¯ ∞ ∪ ¯¯ ∞ ∪ ¯¯ ∞ , which are defined as (see Figure 2 ε is a positive
1
constant, used later)
2
3
5
ε tπ i 1
e ≤t ≤ ,
¯¯ ∞
=
2
2
3
3
ε
π i/3 ¯¯ ∞
−∞ < t ≤ − ,
1 = −te
2
ε
5π i/3 ¯¯ ∞
≤t <∞ .
3 = te
2
R EMARK 1. The kernel in (5) is not in trace class, but the Fredholm determinant is well defined and we can easily conjugate it into a trace class kernel. Several
kernels below are in similar situations.
F IG . 2.
¯¯ ∞ .
1277
QUATERNIONIC WISHART
In the real category, we have the result for the real white Whishart ensemble:
P ROPOSITION 3. The distribution of the largest sample eigenvalue in the real
white Wishart ensemble satisfies that, max(λ) almost surely approaches [15] (1 +
γ −1 )2 with fluctuation scale M −2/3 , and [19]
lim P max(λ) − (1 + γ
−1 2 γ M 2/3
) ·
≤T
(1 + γ )4/3
M→∞
= FGOE (T ),
where FGOE is the GOE Tracy–Widom distribution.
Here the function FGOE is defined by the Fredholm determinant of a matrix
integral operator [30]:
FGOE (T ) = det I − PGOE (ξ, η)
and
PGOE (ξ, η) = χ(T ,∞) (ξ )
S1 (ξ, η)
SD1 (ξ, η)
χ(T ,∞) (η),
1
IS1 (ξ, η) − 2 sgn(x − y) S1 (η, ξ, )
where
S1 (ξ, η) = KAiry (ξ, η) −
SD1 (ξ, η) = −
IS1 (ξ, η) = −
(6)
∞
Ai(t) dt +
η
∂
1
KAiry (ξ, η) − Ai(ξ ) Ai(η),
∂η
2
∞
−
R EMARK 2.
1
Ai(ξ )
2
ξ
1
2
KAiry (t, η) dt +
∞
ξ
Ai(t) dt +
1
2
1
2
∞
Ai(t) dt
ξ
∞
1
Ai(ξ ),
2
∞
Ai(t) dt
η
Ai(t) dt.
η
We have a more convenient form of FGOE [12]:
FGOE = det 1 − KAiry (ξ, η) + s (1) (ξ ) Ai(η) χ(T ,∞) (η) ,
so [4]
FGUE1 (T ) = (FGOE (T ))2 .
In the real spiked model, Baik and Silverstein [8] compute the almost sure limit
of the largest population eigenvalue, which is the same as that in the complex
category, and Paul [25] proves the Gaussian distribution property in the case a >
γ −1 , which is similar to (4). Neither of their methods can find the distribution
function when a ≤ γ −1 .
For the quaternionic white Wishart ensemble, we have:
1278
D. WANG
P ROPOSITION 4. The distribution of the largest sample eigenvalue in the
quaternionic white Wishart ensemble satisfies that, max(λ) almost surely approaches (1 + γ −1 )2 with fluctuation scale M −2/3 , and [14]
−1 2 γ (2M)2/3
≤T
lim P max(λ) − (1 + γ ) ·
M→∞
(1 + γ )4/3
where FGSE is the GSE Tracy–Widom distribution.
= FGSE (T ),
Here the function FGSE is defined by the Fredholm determinant of a matrix
integral operator [30]:
FGSE (T ) = det I − P(ξ, η)
and
4 (ξ, η)
4 (ξ, η) SD
S
P̂ (ξ, η) = χ(T ,∞) (ξ ) 4 (η, ξ, ) χ(T ,∞) (η),
IS4 (ξ, η) S
where
4 (ξ, η) = 1 KAiry (ξ, η) − 1 Ai(ξ )
S
2
4 (ξ, η) = −
SD
4
∞
Ai(t) dt,
η
1 ∂
1
KAiry (ξ, η) − Ai(ξ ) Ai(η),
2 ∂η
4
4 (ξ, η) = − 1
IS
2
∞
ξ
1
KAiry (t, η) dt +
4
∞
Ai(t) dt
∞
ξ
Ai(t) dt.
η
1.2. Statement of main results. The main theorem in this paper is:
T HEOREM 1.
In the rank 1 quaternionic spiked model:
1. If −1 < a < γ −1 , then the distribution of the largest sample eigenvalue is the
same as that of the quaternionic white Wishart ensemble in Proposition 4.
2. If a = γ −1 , then the limit and the fluctuation scale are the same as those of the
quaternionic white Wishart ensemble, but the distribution function is
lim P
M→∞
max(λ) −
γ +1
γ
2 ·
γ (2M)2/3
≤T
(1 + γ )4/3
= FGSE1 (T ).
3. If a > γ −1 , then the limit and the fluctuation scale are changed as well as the
distribution function, which is a Gaussian:
√
2M
1
≤T
·
lim P max(λ) − (a + 1) 1 + 2
M→∞
γ a
(a + 1) 1 − 1/(γ 2 a 2 )
=
T
1
2
√ e−t /2 dt.
−∞ 2π
1279
QUATERNIONIC WISHART
Here the function FGSE1 is defined by the Fredholm determinant of a matrix
integral operator:
FGSE1 (T ) = det I − P (ξ, η)
and
S 4 (ξ, η) SD4 (ξ, η)
P (ξ, η) = χ(T ,∞) (ξ )
χ(T ,∞) (η),
IS4 (ξ, η) S 4 (η, ξ, )
where
1
4 (ξ, η),
Ai(ξ ),
SD4 (ξ, η) = SD
2
∞
1 ∞
4 (ξ, η) − 1
IS4 (ξ, η) = IS
Ai(t) dt +
Ai(t) dt.
2 ξ
2 η
4 (ξ, η) +
S 4 (ξ, η) = S
Although the distribution FGSE1 seems to be new, we have that
T HEOREM 2.
FGSE1 (T ) = FGOE (T ).
1.3. Relation with other models and conjecture on the rank 1 real spiked
model. The results of Theorems 1 and 2 give a phase transition pattern FGSE –
FGOE –Gaussian as the parameter a increases from −1 to +∞. This pattern appears as limiting distributions indexed by a parameter in several other combinatorial and statistical physical models, for example, the lengths of the longest
monotone subsequences of random involutions with condition on the number of
fixed points [6] and the symmetrized last passage percolation [7] studied by Baik
and Rains. In semi-infinite totally asymmetric simple exclusion process [26] studied by Prähofer and Spohn, and the symmetric polynuclear growth process [5]
studied by Baik et al., 2-dimensional phase transition diagrams are obtained, and
the 1-dimensional FGSE –FGOE –Gaussian pattern is contained in both of them.
Although there is no model which can give hints to the rank 1 real spiked model,
it is plausible that it has a phase transition from FGOE to Gaussian for the limiting
distributions of the largest sample eigenvalue as a goes across γ −1 . Based on the
duality of orthogonal and symplectic models from the Virasoro structure’s point of
view, we have:
C ONJECTURE 1.
In the rank 1 real spiked model:
1. If −1 < a < γ −1 , then the distribution of the largest sample eigenvalue is the
same as that of the real white Wishart ensemble in Proposition 3.
1280
D. WANG
2. If a = γ −1 , then the limit and the fluctuation scale are the same as those of the
quaternionic white Wishart ensemble, but the distribution function is
lim P
M→∞
γ +1
max(λ) −
γ
2 γ (M/2)2/3
·
≤T
(1 + γ )4/3
= FGSE (T ).
3. If a > γ −1 , then the limit and the fluctuation scale are changed as well as the
distribution function, which is a Gaussian (proved by Paul in [25]):
√
M/2
1
≤T
·
lim P max(λ) − (a + 1) 1 + 2
M→∞
γ a
(a + 1) 1 − 1/(γ 2 a 2 )
=
T
1
2
√ e−t /2 dt.
−∞ 2π
1.4. Structure of the paper. In Section 2 we use combinatorial techniques to
express the joint distribution function of {λj }, and then by skew orthogonal polynomial techniques express the distribution function of max(λj ) in the square root
of a Fredholm determinant of a matrix integral operator. In Section 3 we do asymptotic analysis on the kernel of the matrix integral operator, and prove the three
cases of Theorem 1 in the three subsections, respectively. Section 4 contains the
proof of Theorem 2. In the proof of Theorem 1, we use some trace norm convergence results which generalize the old result on the LUE [11], and we give a
method of proof to them in the Appendix.
2. The Fredholm determinantal formula.
2.1. The joint distribution function. In this subsection, we prove the following:
T HEOREM 3. The joint probability distribution function of λ in the quaternionic spiked model is
(7)
P (λ) =
N
2(M−N)+1 −2Mλ 1 4
j .
λj
e
Ṽ (λ)
C
j =1
In this paper, C stands for any constants, and here
1
λ
1
λ21
..
Ṽ 4 (λ) = .
2N −2
λ
1
ea/(1+a)2Mλ1
0
1
2λ1
..
.
−3
(2N − 2)λ2N
1
···
···
···
···
···
λ2N−2
N
a
a/(1+a)2Mλ1
1+a 2Me
···
ea/(1+a)2MλN
1
λN
λ2N
..
.
,
a 2Mea/(1+a)2MλN 1+a
0
1
2λN
..
.
(2N − 2)λ2N−3
N
1281
QUATERNIONIC WISHART
the determinant of a 2N × 2N matrix whose (2N, 2k − 1) entry is ea/(1+a)2Mλk ,
j −1
(j, 2k − 1) entry is λk for j = 1, . . . , 2N − 1, and 2ith column is the derivative
of the (2i − 1)st column. Ṽ 4 (λ) is a variation of the V (λ)4 appearing in the LSE
(see [23] and (9)).
For the Wishart ensemble defined in the introduction section, we first have the
distribution function for the sample matrix in the N × N positive definite quaternionic Hermitian matrix space [3]:
P (S) =
1 −2M Tr( −1 S)
(det S)2(M−N)+1 .
e
C
R EMARK 3. Due to the noncommutativity of the quaternions, det S is not well
defined in the usual way. Since S is quaternionic Hermitian, we can diagonalize
it into a real-valued diagonal matrix by the conjugation of a quaternionic unitary
matrix U , and define
det S =
eigenvalues of U SU ∗ .
N
R EMARK 4. In the distribution function in real and complex categories of
sample matrices, we do not need to take the real part of the trace, since the trace is
already real. Unfortunately, this does not hold in the quaternionic category due to
its noncommutativity, and luckily Tr behaves better. [For example, Tr(AB) =
Tr(BA), but Tr(AB) = Tr(BA) in general.]
The distribution function for sample eigenvalues λ, the eigenvalues of S, is
(8)
N
1
λj2(M−N)+1
P (λ) = (V (λ))4
C
j =1
e−2M Tr(
−1 QQ−1 )
dQ,
Q∈Sp(N)
where we
integrate on the compact symplectic group with the Haar measure,
V (λ) = i<j (λi − λj ) is the Vandermonde, and = diag(λ1 , . . . , λN ). (See [23]
for a derivation of the similar GSE case.)
If the perturbation parameter a = 0, then l1 = l2 = · · · = lN = 1,
e−2M Tr(
Q∈Sp(N)
−1 QQ−1 )
dQ =
N
e−2Mλj
j =1
and
(9)
P (λ) =
is the standard LSE [23].
N
2(M−N)+1 −2Mλ 1
j ,
λj
e
(V (λ))4
C
j =1
1282
D. WANG
Generally,
e−2M Tr(
−1 QQ−1 )
dQ
Q∈Sp(N)
=
(10)
=
−1 )
e−2M Tr(I QQ
e−2M Tr((
−1 −I )QQ−1 )
dQ
Q∈Sp(N)
N
e−2Mλj
e2M Tr((I −
−1 )QQ−1 )
dQ.
Q∈Sp(N)
j =1
Then by the integral formula of the quaternionic Zonal polynomials [17], we get
e2M Tr((I −
−1 )QQ−1 )
dQ
Q∈Sp(N)
(11)
=
∞
(2M)j
j!
j =0
Cκ(1/2) (I − −1 )Cκ(1/2) ()
(1/2)
Cκ
l(κ)≤N
κj
,
(IN )
(1/2)
where Cκ (x1 , . . . , xN ) is the N variable quaternionic Zonal polynomial, that is,
the Jack polynomial with the parameter α = 1/2 (see [21] and [27]) and the Cnormalization [10], so that [κ = (k1 , . . . , kl ), k1 ≥ k2 ≥ · · · ≥ kl > 0, then l(κ) = l]
Cκ(1/2) (x1 , . . . , xm ) = (x1 + · · · + xm )k .
l(κ)≤m
κk
In the formula, a symmetric polynomial of a matrix is equivalent to the symmetric
polynomial of its eigenvalues, so
Cκ(1/2) (I − −1 ) = Cκ(1/2)
a
, 0, . . . , 0 .
1+a
(1/2)
Since all variables except for one vanish in Cκ
(I − −1 ), we simply find
Cκ(1/2) (I − −1 )l(κ)>1 = 0.
(12)
We have [27]
(1/2)
C(j )
a
a
, 0, . . . , 0 =
1+a
1+a
j
and since the number of variables is N [27]
(1/2)
C(j ) (1, . . . , 1) =
j
−1
1
(2N + i),
(j + 1)! i=0
1283
QUATERNIONIC WISHART
so with (11) and (12), we get
e2M Tr((I −
−1 )QQ−1 )
dQ
Q∈Sp(N)
−1
∞
(2M)j C(j ) (I − )C(j ) ()
(1/2)
=
j!
j =0
∞
(1/2)
(1/2)
C(j ) (IN )
j +1
a
2M
=
j −1
1+a
j =0 i=0 (2N + i)
j
(1/2)
C(j ) ().
In [27] there is an identity
∞
(1/2)
(j + 1)C(j ) ()t j =
j =0
N
1
.
(1 − λj t)2
j =1
Comparing it with the well-known identity for Schur polynomials
∞
s(j ) ()t j =
j =0
N
1
,
1 − λj t
j =1
we get the identity
(1/2)
(j + 1)C(j ) () = s(j ) (λ1 , λ1 , λ2 , λ2 , . . . , λN , λN ),
(13)
with each λi appearing twice as variables of the s(j ) . For notational simplicity, we
denote the right-hand side of (13) as s̃(j ) (), which is a plethysm [21]
s̃(j ) () = s(j ) ◦ 2p1 ().
Now we get
e2M Tr((I −
−1 )QQ−1 )
dQ
Q∈Sp(N)
(14)
∞
1
a
2M
=
j −1
1+a
j =0 i=0 (2N + i)
j
s̃(j ) ().
Then we need a lemma to simplify (14) further.
L EMMA 1.
(15)
1
0
λ
1
1
2λ1
λ21
..
s̃(j ) () = ..
.
.
2N−2
2N−3
λ1
(2N
−
2)λ
1
λ2N+j −1 (2N + j − 1)λ2N+j −2
1
1
× V (λ)
−4
,
···
···
···
···
···
···
1
λN
λ2N
..
.
λ2N−2
N
2N+j −1
λN
2N+j −2 (2N + j − 1)λ
0
1
2λN
..
.
(2N − 2)λ2N−3
N
N
1284
D. WANG
with the (k, 2j − 1) entry of the matrix being a power of λj with the exponent k − 1
if k = 2N and 2N + j − 1 if k = 2N , and the (k, 2j ) entry being the derivative of
the (k, 2j − 1) entry with respect to λj .
To prove this lemma, we need the well-known fact (see [23]), proven by
L’Hôpital’s rule
1
λ1
4
V (λ) = ..
.
2N−1
λ
(16)
1
···
···
0
1
..
.
(2N
1
λN
..
.
0
1
..
.
···
2N−1
· · · λN
− 1)λ12N−2
,
2N−2 (2N − 1)λN
with the (k, 2j − 1) entry being λk−1
and the (k, 2j ) entry (k − 1)λk−2
j
j .
P ROOF OF L EMMA 1. Applying the L’Hôpital’s rule repeatedly with respect
to x2i , i = 1, . . . , N , we get the identity
1
λ1
0
1
···
···
1
λN
λ21
.
.
.
2λ1
.
.
.
···
···
λ2N
.
.
.
−2
λ2N
1
−3
(2N − 2)λ2N
1
···
−2
λ2N
N
2N +j −1
λ1
(2N + j
2N +j −2
− 1)λ1
···
2N +j −1
0
1
.
.
.
···
···
···
1
λN
.
.
.
−1
λ2N
1
(2N − 1)λ2N −2
···
−1
λ2N
N
N
⎜
∂
⎜
=⎜
⎝ ∂x2 ∂x4 · · · ∂x2N (2N + j − 1)λN
λN
1
λ1
.
.
.
⎛
2λN
.
.
.
−3
(2N − 2)λ2N
N
2N +j −2
0
1
2N −2 0
1
.
.
.
(2N − 1)λN
1
x1
.
.
.
1
x2
.
.
.
···
···
···
1
x2N −1
.
.
.
x12N −2
x22N −2
···
2N +j −1
x1
2N +j −1
x2
2N −2
x2N
−1
N
∂
∂x2 ∂x4 · · · ∂x2N 2N +j −1
· · · x2N −1
1
x1
.
.
.
1
x2
.
.
.
···
···
···
1
x2N −1
.
.
.
x12N −2
x22N −2
···
2N +j −1
x1
2N +j −1
x2
2N −2
x2N
−1
···
x2N
2N +j −1
x2N −1
2N −2 x2N
2N +j −1
1
x2N
.
.
.
⎞
1
⎟
⎟
x2N ⎟
⎟
.
.
⎟
.
⎟
2N −2
⎟
x2N
⎟
2N +j −1 ⎠
x2N
x2i−1 = x2i = λi
i = 1, . . . , N
= s(j ) (λ1 , λ1 , λ2 , λ2 , . . . , λN , λN ) = s̃(j ) (),
from the matrix representation of Schur polynomials, and now use (16) to get the
compact formula (15). 1285
QUATERNIONIC WISHART
Substituting (15) into (14), we get
4
e2M Tr((I −
V (λ)
−1 )QQ−1 )
Q∈Sp(N)
0
···
1
1
λ1
1
···
λN
λ2
2λ1
···
λ2N
1
.
.
= ..
.
.
.
.
···
.
λ2N −2 (2N − 2)λ2N −3 · · · λ2N −2
1
N
1
p(λ )
p (λ1 )
· · · p(λN )
1
1
0
λ1
1
2
λ
2λ
1
1
1
.
.
= .
.
.
.
C
λ2N −2
−3
(2N − 2)λ2N
1
1
ea/(1+a)2Mλ1 a 2Mea/(1+a)2Mλ1
(17)
1+a
=
dQ
2λN
.
.
.
2N −3 (2N − 2)λN
p (λ )
0
1
N
···
···
1
λN
0
1
···
λ2N
.
.
.
2λN
.
.
.
−2
λ2N
N
a/(1+a)2Mλ
N
e
−3
(2N − 2)λ2N
N
a
a/(1+a)2Mλ
N
1+a 2Me
···
···
···
1 4
Ṽ (λ),
C
where
∞
1
a
2M
p(x) =
j −1
1+a
j =0 i=0 (2N + i)
j
x 2N+j −1
2N−2
1
(2N − 1)!
a
a/(1+a)2Mx
=
e
−
2Mx
2N−1
(a/(1 + a)2M)
j! 1 + a
j =0
j ,
and if k = 2N , the (k, 2j − 1) entries in both matrices are λk−1
j , and the (k, 2j )
k−2
entries are (k − 1)λj , and the 2N, 2i − 1 entry in the former (latter) matrix is
p(λi ) (resp. ea/(1+a)2Mλi ) and the 2N, 2i entry p (λi ) (resp.
P ROOF OF T HEOREM 3.
sult (7). a
a/(1+a)2Mλi ).
1+a 2Me
Formulas (8), (10) and (17) together give the re-
2.2. The Pfaffian and determinantal formulas. With the formula (7) ready to
use, we apply the standard RMT technique to get the distribution formula for the
largest sample eigenvalue, in the same spirit as the solution of the LSE. Our process
below is closely parallel to that in [31] to the LSE.
First, we find a skew orthogonal basis {ϕ0 (x), ϕ1 (x), . . . , ϕ2N−1 (x)} of the linear space spanned by {1, x, x 2 , . . . , x 2N−2 , ea/(1+a)2Mx }. We require that the ϕj (x)
is a linear combination of {1, x, x 2 , . . . , x j } if j < 2N − 1, while ϕ2N−1 (x) can be
1286
D. WANG
arbitrary, with the skew inner products among them
ϕj (x), ϕk (x)4 =
=
∞
0
ϕj (x)ϕk (x) − ϕj (x)ϕk (x) x 2(M−N)+1 e−2Mx dx
⎧
⎨ rj/2 ,
⎩
if j is even and k = j + 1,
if k is even and j = k + 1,
otherwise.
−rk/2 ,
0,
Then we can reformulate the distribution function of λ as
1 P (λ) = C ×
(18)
ϕ0 (λ1 )
ϕ1 (λ1 )
..
.
ϕ2N−1 (λ1 )
ϕ0 (λ1 )
ϕ1 (λ1 )
..
.
ϕ2N−1
(λ1 )
···
···
ϕ0 (λN )
ϕ1 (λN )
..
.
ϕ0 (λN )
ϕ1 (λN )
..
.
···
· · · ϕ2N−1 (λN ) ϕ2N−1
(λN )
N
2(M−N)+1 −2Mλ j
λ
e
j =1
1 = C j
ψ0 (λ1 )
ψ1 (λ1 )
..
.
ψ2N−1 (λ1 )
ψ0 (λ1 )
ψ1 (λ1 )
..
.
ψ2N−1
(λ1 )
···
···
ψ0 (λN )
ψ1 (λN )
..
.
···
· · · ψ2N−1 (λN )
ψ0 (λN )
ψ1 (λN )
..
.
ψ2N−1
(λN )
,
where
ψi (x) = ϕi (x)x M−N+1/2 e−Mx .
(19)
For an arbitrary function f (x) on [0, ∞), by the formula of de Bruijn [9],
∞
∞ ···
0
0 (20)
ψ0 (λ1 )
ψ1 (λ1 )
..
.
ψ2N−1 (λ1 )
×
N
ψ0 (λ1 )
ψ1 (λ1 )
..
.
ψ2N−1
(λ1 )
···
···
ψ0 (λN )
ψ1 (λN )
..
.
···
· · · ψ2N−1 (λN )
ψ0 (λN )
ψ1 (λN )
..
.
ψ2N−1
(λN )
1 + f (λi ) dλi = C Pf P (1 + f ) ,
i=1
where P (1 + f ) is a 2N × 2N matrix, whose entries depend on 1 + f in the
following way:
P (1 + f )
j,k =
∞
0
ψj −1 (x)ψk−1
(x) − ψj −1 (x)ψk−1 (x) 1 + f (x) dx.
1287
QUATERNIONIC WISHART
Now we define a matrix Z as
⎛
0
⎜ −r0
⎜
⎞
r0
0
⎜
⎜
⎜
Z=⎜
⎜
⎜
⎜
⎝
0
−r1
r1
0
..
.
0
−rN−1
with
Zj,k =
⎧
⎨ rk/2−1 ,
,
−r
⎩ j/2−1
0,
rN−1
0
⎟
⎟
⎟
⎟
⎟
⎟,
⎟
⎟
⎟
⎠
if k is even and j = k − 1,
if j is even and k = j − 1,
otherwise,
and define for j = 0, . . . , N − 1, η = Z −1 ψ, that is,
η2j (x) = −
So we have
P (1 + f )
j,k
=
ψ2j +1 (x)
rj
and
η2j +1 (x) =
ψ2j (x)
.
rj
∞
0
+
ψj −1 (x)ψk−1
(x) − ψj −1 (x)ψk−1 (x) dx
∞
0
= Zj,k +
ψj −1 (x)ψk−1
(x) − ψj −1 (x)ψk−1 (x) f (x) dx
∞
0
ψj −1 (x)ψk−1
(x) − ψj −1 (x)ψk−1 (x) f (x) dx.
And if we denote Q(1 + f ) = Z −1 P (1 + f ), then
Q(1 + f )j,k = δj,k +
∞
0
ηj −1 (x)ψk−1
(x) − ηj −1 (x)ψk−1 (x) f (x) dx.
If we choose f to be −χ(T ,∞) , then the integral on the left-hand side of (20),
after multiplying a constant, is the probability of all λi ’s smaller than T . In latter part of the paper, we abbreviate χ(T ,∞) to χ . So we get for a T -independent
constant
P max(λi ) ≤ T = C Pf P (1 − χ ) ,
and
P max(λi ) ≤ T
2
In linear algebra, we have the determinant identity
(21)
= C 2 det P (1 − χ ) = C 2 det Q(1 − χ ) .
det(I − AB) = det(I − BA),
1288
D. WANG
for A an m × n matrix and B an n × m matrix, but the identity still holds in infinite
dimensional settings [16]. Letting det mean a Fredholm determinant for matrix
integral operators, we describe a setting due to Tracy–Widom [31].
If A is an operator from L2 ([0, ∞)) × L2 ([0, ∞)) to the vector space R2N with
A
g(x)
h(x)
j
=
∞
0
χ (x)ηj −1 (x)g(x) dx −
∞
0
χ (x)ηj −1 (x)h(x) dx,
and B is an operator from R2N to L2 ([0, ∞)) × L2 ([0, ∞)) with
⎛
⎛ 2N
⎞
ck ψk−1 (x)χ(x) ⎟
⎜
⎟
⎟ ⎜
k=1
⎜
⎟,
⎠ = ⎜ 2N
⎟
⎝
⎠
⎞
c1
⎜ .
B ⎝ ..
c2N
ck ψk−1 (x)χ(x)
k=1
then
I − AB = Q(1 − χ )
and
S4 (x, y) SD4 (x, y)
I − BA = I − χ (x)
χ (y),
IS4 (x, y) S4 (y, x)
where S4 (x, y), IS4 (x, y) and SD4 (x, y) are integral operators whose kernels are
S4 (x, y) =
2N−1
j =0
=
N−1
j =0
SD4 (x, y) =
1
−ψ2j
(x)ψ2j +1 (y) + ψ2j
+1 (x)ψ2j (y) ,
rj
2N−1
j =0
(22)
=
N−1
j =0
IS4 (x, y) =
ψj (x)ηj (y)
−ψj (x)ηj (y)
1 ψ2j (x)ψ2j
+1 (y) − ψ2j +1 (x)ψ2j (y) ,
rj
2N−1
ψj (x)ηj (y)
j =0
=
N−1
j =0
1
−ψ2j (x)ψ2j +1 (y) + ψ2j +1 (x)ψ2j (y) ,
rj
1289
QUATERNIONIC WISHART
S4 (y, x) =
2N−1
j =0
(23)
=
N−1
j =0
−ψj (x)ηj (y)
1
ψ2j (x)ψ2j
+1 (y) − ψ2j +1 (x)ψ2j (y) .
rj
R EMARK 5. It is clear that the nomenclature of SD4 (x, y) is due to the fact
that SD4 (x, y) is the negative of the derivative of S4 (x, y). But IS4 (x, y), which
gets its name in the same way in earlier literature in GSE (e.g., [30]), in our problem may not satisfy the equation
IS4 (x, y) = −
∞
x
S4 (t, y) dt,
since the integral on the right-hand side may diverge.
In conclusion,
2
P max(λi ) ≤ T
S4 (x, y) SD4 (x, y)
= C det I − χ (x)
χ (y) ,
IS4 (x, y) S4 (y, x)
2
and we can find that C 2 = 1 by taking the limit T → ∞. We define a 2 × 2 matrix
kernel as
PT (x, y) = χ (x)
=
S4 (x, y) SD4 (x, y)
χ (y)
IS4 (x, y) S4 (y, x)
χ (x)S4 (x, y)χ(y) χ(x)DS4 (x, y)χ(y)
,
χ (x)IS4 (x, y)χ(y) χ(x)S4 (y, x)χ(y)
then we have
P max(λi ) ≤ T
2
= det I − PT (x, y) .
2.3. S4 (x, y) in terms of Laguerre polynomials. In manipulation of skew orthogonal polynomials, we take the approach of [1], and all classical orthogonal
polynomial properties are from [28].
Since Laguerre polynomials by definition satisfy the orthogonal property
∞
0
(α) α −x
L(α)
dx =
j Lk x e
(j + α)!
δj,k ,
j!
(α)
and they have the differential identity [we assume Ln (x) = 0 if n < 0]
(24)
x
d (α)
(α)
L (x) = nL(α)
n (x) − (n + α)Ln−1 (x),
dx n
1290
D. WANG
it is easy to get that
(2(M−N))
Lj
=
(2Mx), L(2(M−N))
(2Mx) 4
k
∞
0
d (2(M−N))
L
(2Mx)
dx k
d (2(M−N))
(2(M−N))
− Lk
(2Mx) Lj
(2Mx)
dx
(2(M−N))
Lj
(2Mx)
× x 2(M−N)+1 e−2Mx dx
⎧
⎪
1 2(M−N)+1 (j + 2(M − N))!
⎪
⎪
,
⎪
⎪
⎨ 2M
(j − 1)!
=
1 2(M−N)+1 (k + 2(M − N))!
⎪
⎪
−
,
⎪
⎪
⎪
2M
(k − 1)!
⎩
0,
if j = k + 1,
if k = j + 1,
otherwise.
So we can choose for j = 0, . . . , N − 2,
ϕ2j (x) =
(25)
j k
k=0
(2(M−N))
ϕ2j +1 (x) = −L2j +1
(26)
2i − 1
(2(M−N))
L2k
(2Mx),
2i
+
2(M
−
N)
i=1
(2Mx)
and
(27)
rj =
1
2M
2(M−N)+1
We can also choose
ϕ2N−2 (x) =
j
(2j + 2(M − N) + 1)! 2k − 1
.
(2j )!
2k + 2(M − N)
k=1
N−1
k
k=0
2i − 1
L(2(M−N))
(2Mx),
2k
2i
+
2(M
−
N)
i=1
but ϕ2N−1 (x) is not a polynomial and needs to be treated separately.
By the Rodrigues’ representation
1 d n −x n+α
(e x
),
n! dx n
and repeated integration by parts, we get for n > 0
x α e−x L(α)
n (x) =
a/(1+a)2Mx (2(M−N))
e
, Ln
(2Mx) 4
2(M−N)+1 =
1+a
2M
(n + 2(M − N) + 1)!
n!
(n + 2(M − N))!
− (−a)n−1
(n − 1)!
(−a)n+1
1291
QUATERNIONIC WISHART
and
a/(1+a)2Mx (2(M−N))
1 + a 2(M−N)+1 e
, L0
(2Mx) 4 = −
a 2(M − N) + 1 !,
2M
so that
a/(1+a)2Mx
e
, ϕ2j (x) 4
=−
and
1+a
2M
2(M−N)+1
a 2j +1
j
(2j + 2(M − N) + 1)! 2k − 1
(2j )!
2k + 2(M − N)
k=1
a/(1+a)2Mx
e
, ϕ2j +1 (x) 4
2(M−N)+1 =−
1+a
2M
a 2j +2
(2j + 2(M − N) + 2)!
(2j + 1)!
(2j + 2(M − N) + 1)!
−a
.
(2j )!
Now by the skew orthogonality, we can choose
2j
ϕ2N−1 (x) = ea/(1+a)2Mx −
N−2
j =0
1 a/(1+a)2Mx
e
, ϕ2j +1 (x) 4 ϕ2j (x)
rj
− ea/(1+a)2Mx , ϕ2j (x) 4 ϕ2j +1 (x)
− (1 + a)2(M−N)+1 a 2N−2
N−1
j =1
=e
a/(1+a)2Mx
− (1 + a)
2j + 2(M − N)
ϕ2N−2 (x)
2j − 1
2(M−N)+1
2N−2
j =0
and
rN−1 =
1+a
2M
2(M−N)+1
a 2N−1
(2(M−N))
(−a)j Lj
(2Mx)
(2M − 1)! N−1
2k − 1
.
(2N − 2)! k=1 2k + 2(M − N)
Now, we write S4 (x, y) as S4a (x, y) + S4b (x, y), where
(28)
S4a (x, y) =
N−2
j =0
1
−ψ2j
(x)ψ2j +1 (y) + ψ2j
+1 (x)ψ2j (y)
rj
and
(29)
S4b (x, y) =
1 −ψ2N−2
(x)ψ2N−1 (y) + ψ2N−1
(x)ψ2N−2 (y) ,
rN−1
and simplify them separately.
1292
D. WANG
The formula (28) of our S4a (x, y) is also the formula for S4 (x, y) in the LSE
problem, with parameters M and N − 2, and has been well studied. For completeness we derive its Laguerre polynomial expression here, following [1].
By the differential identity (24) and the identity
(α)
(α)
nL(α)
n (x) = (−x + 2n + α − 1)Ln−1 (x) − (n + α − 1)Ln−2 (x),
we get, remembering the definition (19), the telescoping sequence
ψ2j
(x) =
j k
k=0
2i − 1
2i + 2(M − N)
i=1
d
(2(M−N))
× M − N + 1/2 − Mx + x
L2k
(2Mx)
dx
× x M−N−1/2 e−Mx
j
=
(30)
k
2i − 1
1
(2(M−N))
(2k + 1)L2k+1
(2Mx)
2 k=0 i=1 2i + 2(M − N)
(2(M−N))
− 2k + 2(M − N) L2k−1
× x M−N−1/2 e−Mx
j
1 2k − 1
=
2 k=1 2k + 2(M − N)
× (2j + 1)L(2(M−N))
(2Mx)x M−N−1/2 e−Mx
2j +1
and
ψ2j
+1 (x) = − M − N + 1/2 − Mx + x
d
(2(M−N))
L
dx 2j +1
× (2Mx)x M−N−1/2 e−Mx
1
(2(M−N))
= − (2j + 2)L2j +2
(2Mx)
2
(31)
(2(M−N))
− 2j + 2(M − N) + 1 L2j
(2Mx)
× x M−N−1/2 e−Mx .
Therefore, if we substitute (27), (30) and (31) into (28), we get after some trick,
1
S4a (x, y) = (2M)2(M−N)+1 x M−N−1/2 e−Mx y M−N+1/2 e−My
2
×
!2N−2
j =0
(2Mx)
j!
(2(M−N))
(2(M−N))
Lj
(2Mx)Lj
(2My)
(j + 2(M − N))!
1293
QUATERNIONIC WISHART
2j + 2(M − N)
(2N − 2)! N−1
−
(2M − 2)! j =1
2j − 1
"
× L(2(M−N))
(2Mx)ϕ2N−2 (y)
2N−2
.
Furthermore, we can simplify ψ2N−2 (x). Since for j = 2N − 1 [if we define
ϕj (x) and then ψj (x) for j > 2N − 1 by the formula (25) and (26)]
∞
0
ψ2N−2 (x)ψj (x) − ψ2N−2
(x)ψj (x) dx = 0,
we get for j = 2N − 1, using integration by parts,
∞
0
ψ2N−2
(x)Lj
(2(M−N))
(2Mx)x M−N+1/2 e−Mx dx = 0.
So by the orthogonal property of Laguerre polynomials, we get
ψ2N−2
(x) = CL(2(M−N))
(2Mx)x M−N−1/2 e−Mx ,
2N−1
and we can determine that
2j − 1
2N − 1 N−1
2
2j + 2(M − N)
j =1
C=
without much difficulty. Together with the fact limx→∞ ψ2N−2 (x) = 0, we get
ψ2N−2 (x) = −
2j − 1
2N − 1 N−1
2
2j + 2(M − N)
j =1
×
∞
x
t M−N−1/2 e−Mt L(2(M−N))
(2Mt) dt.
2N−1
Now, we can write S4a (x, y) as S4a1 (x, y) + S4a2 (x, y), where
1
S4a1 (x, y) = (2M)2(M−N)+1
2
(32)
×
2N−2
j =0
j!
(2(M−N))
Lj
(2Mx)x M−N−1/2 e−Mx
(j + 2(M − N))!
(2My)y M−N+1/2 e−My
× L(2(M−N))
j
and
1
(2N − 1)!
S4a2 (x, y) = (2M)2(M−N)+1
4
(2M − 2)!
(33)
(2(M−N))
× L2N−2
×
∞
y
(2Mx)x M−N−1/2 e−Mx
t M−N−1/2 e−Mt L(2(M−N))
(2Mt) dt.
2N−1
1294
D. WANG
Finally,
1 2M
S4b (x, y) = −
2 1+a
(2(M−N))
× L2N−1
(34)
2(M−N)+1
a −(2N−1)
(2N − 1)!
(2M − 1)!
(2Mx)x M−N−1/2 e−Mx ψ2N−1 (y)
+ ψ2N−1
(x)
∞
y
(2(M−N))
L2N−1
(2Mt)t M−N−1/2 e−Mt dt ,
and we can take the asymptotic analyses of S4a1 (x, y), S4a2 (x, y) and S4b (x, y)
separately.
3. Asymptotic analysis. In order to consider the rescaled distribution problem, we wish to find the probability of the largest sample eigenvalue being in the
domain (0, p + qT ]. We can put the kernel in the new coordinate system [after a
conjugation by
q 1/2
0
0
],
q −1/2
P max(λi ) ≤ p + qT
2
and get
\$ 4 (ξ, η)
# (ξ, η) SD
S
= det I − χ (x) #4
#4 (η, ξ ) χ (η)
IS4 (ξ, η) S
= det I − P#T (ξ, η) ,
where as L2 functions,
\$ 4 (ξ, η) = q 2 SD4 (x, y)|x=p+qξ ,
SD
(35)
y=p+qη
#4 (ξ, η) = qS4 (x, y)|x=p+qξ ,
S
(36)
y=p+qη
# 4 (ξ η) = IS4 (x, y)|x=p+qξ
IS
(37)
y=p+qη
and
\$ 4 (ξ, η)
# (ξ, η) SD
S
P#T (ξ, η) = χ (ξ ) #4
#4 (η, ξ ) χ (η).
IS4 (ξ, η) S
In this section, we want to prove that for fixed γ ≥ 1 and a > −1, we can choose
suitable pM and qM depending on M, so that for any T ,
lim P max(λi ) ≤ pM + qM T
M→∞
2
= lim det I − P#T (ξ, η) = fa (T ),
M→∞
where fa is a function to be determined.
To prove the convergence of Fredholm determinants, we may use that P#T (ξ, η)
is in trace class for any M and converges to a certain 2 × 2 matrix kernel in trace
norm. Equivalently, we may use that each entry of P#T (ξ, η) is in trace class and
converges to a scalar kernel in trace norm. It turns out later that the P#T (ξ, η)’s may
not satisfy these requirements, but certain conjugates do.
1295
QUATERNIONIC WISHART
Since the IS4 (x, y) and DS4 (x, y) are of the same form as S4 (x, y), we only
show the asymptotic analysis of S4 (x, y), and state the result for the other two, for
which the arguments are the same.
3.1. Proof of the −1 < a < γ −1 part of Theorem 1. In case −1 < a ≤ γ −1 ,
)4/3
, and denote [here ∗ stands for 4,
we choose pM = (1 + γ −1 )2 and qM = γ(1+γ
(2M)2/3
#∗ (ξ, η) in (38) is only used in Sections 3.1
4a, 4a1, 4a2 and 4b; the definition of S
and 3.2]
(38)
#∗ (ξ, η) =
S
(1 + γ )4/3
S∗ (x, y)|x=(1+γ −1 )2 +(1+γ )4/3 /(γ (2M)2/3 )ξ .
γ (2M)2/3
y=(1+γ −1 )2 +(1+γ )4/3 /(γ (2M)2/3 )η
S4a (x, y) is the formula for the upper-left entry of the 2 × 2 matrix kernel of the
LSE problem with parameters M and N − 1, and its asymptotic behavior is well
studied [14]. We want to prove that as M → ∞, S4a (x, y) dominates S4 (x, y) in
the domain that we are interested in, and so naturally the distribution of the largest
sample eigenvalue in the perturbed problem is the same as that in the LSE problem.
(The difference between N and N − 1 is negligible.)
S4a1 (x, y) is almost the kernel for
√ the LUE problem with parameters
2M − 2 and 2N − 2, besides a factor y/x/2. From a standard result for LUE
#4a1 (ξ, η)χT (η) is in trace class and converges in trace norm to half of
[11], χT (ξ )S
the Airy kernel
#4a1 (ξ, η)χ(η) = 1 χ (ξ )KAiry (ξ, η)χ(η).
lim χ (ξ )S
2
More discussion see the Appendix.
For the S4a2 (x, y) part, we also have in trace norm [14],
(39)
(40)
M→∞
#4a2 (ξ, η)χ(η) = − 1 χ (ξ ) Ai(ξ )
lim χ (ξ )S
4
M→∞
∞
Ai(t) dt χ(η).
η
#4a2 (ξ, η) is a rank 1 operator, for the trace norm
We just sketch the proof. Since S
convergence, we only need to prove that in L2 norm as functions in ξ and respectively η,
lim γ −2N (1 + γ )4/3 (2M)1/3 eM−N
M→∞
(2(M−N))
× L2N−2
(41)
(2Mx)x M−N−1/2 e−Mx χ (ξ )
= Ai(ξ )χ(ξ ),
lim γ
−2N
M→∞
(42)
=−
2Me
∞
η
M−N
∞
y
L(2(M−N))
(2Mt)t M−N−1/2 e−Mt dt χ(η)
2N−1
Ai(t) dt χ(η)
1296
D. WANG
and by the Stirling’s formula,
lim (2M)2(M−N)−1
M→∞
(2N − 1)! 2(N−M) 4N−1
e
γ
= 1.
(2M − 2)!
By (33), (38), (41) and (42), we get
#4a2 (ξ, η)χ(η) = 1 (2M)2(M−N)−1 (2N − 1)! e2(N−M) γ 4N−1 γ −2N
χ (ξ )S
4
(2M − 2)!
× (1 + γ )4/3 (2M)1/3 eM−N L(2(M−N))
(2Mx)
2N−2
× x M−N−1/2 e−Mx χ (ξ )γ −2N 2MeM−N
×
∞
y
(2(M−N))
L2N−1
(2Mt)t M−N−1/2 e−Mt dt χ(η).
Therefore we get the trace norm convergence from the L2 convergence by the fact
that if fn (x) → f (x) and gn (y) → g(y) in L2 norm, then we have the convergence
of integral operators in trace norm:
fn (x)gn (y) → f (x)g(y).
Finally, we need to analyze the term S4b (ξ, η), new to the perturbed problem.
We need the following results:
P ROPOSITION 5. For fixed γ ≥ 1 and −1 < a < γ −1 and any T , we have the
convergences in L2 norm with respect to ξ or η:
lim γ −2N−1 (1 + γ )4/3 (2M)1/3 eM−N
M→∞
(2(M−N))
× L2N−1
(2Mx)x M−N−1/2 e−Mx χ (ξ )
= − Ai(ξ )χ(ξ ),
lim γ −2N 2MeM−N
M→∞
=−
∞
∞
y
(2(M−N))
L2N−1
Ai(t) dt χ(η),
η
lim (1 + a)2(N−M)−1 a −2N+1
(43)
(2Mt)t M−N−1/2 e−Mt dt χ(η)
M→∞
(1 − aγ )(2M)1/3 M−N
e
ψ2N−1 (y)χ(η)
(γ + 1)2/3 γ 2N−1
= Ai(η)χ(η),
lim (1 + a)2(N−M)−1 a −2N+1
M→∞
= Ai (ξ )χ(ξ ).
(1 − aγ )(γ + 1)2/3 M−N e
ψ2N−1 (x)χ(ξ )
γ 2N (2M)1/3
1297
QUATERNIONIC WISHART
P ROOF. We just prove the identity (43), and others can be done in the same
way.
By the integral representation of Laguerre polynomials,
%
e−2Mxz (z + 1)n+2(M−N)
1
dz,
2πi C
zn+1
where C is a contour around the pole 0, therefore we get
L(2(M−N))
(2Mx) =
n
(44)
ϕ2N−1 (y) = ea/(1+a)2My
−
(1 + a)2(M−N)+1
2πi
×
%
e−2Myz
C
(45)
= ea/(1+a)2My −
−
(1 + a)2(M−N)
2πi
%
e−2Myz
C
(z + 1)2(M−N)
dz
z + a/(a + 1)
(1 + a)2(M−N)+1 a 2N−1
2πi
×
If the pole
1 + (a(z + 1)/z)2N−1 (z + 1)2(M−N)
dz
1 + a(z + 1)/z
z
a
z = − a+1
%
e−2Myz
C
(z + 1)2M
z
dz.
2N
z
((a + 1)z + a)(z + 1)
is inside of C, then
%
(1 + a)2(M−N)
2πi
e−2Myz
C
(z + 1)2(M−N)
dz = ea/(a+1)2My
z + a/(a + 1)
and
ϕ2N−1 (y) = −
(46)
(1 + a)2(M−N)+1 a 2N−1
2πi
×
%
C
e−2Myz
(z + 1)2M
z
dz.
2N
z
((a + 1)z + a)(z + 1)
In later part of the proof, we make this condition hold, and will not mention the
canceled terms, and we are then free to deform C in (46) as we wish, provided it
includes 0. We then proceed to a stationary phase analysis.
Since
4/3
(z + 1)2M
2M(−(1+γ −1 )2 z+log(z+1)−γ −2 log z)− (1+γγ) (2M)1/3 ηz
=
e
z2N
(we do not need to concern ourselves about the ambiguity of the value of logarithmic functions), if we denote
e−2Myz
(47)
f (z) = −(1 + γ −1 )2 z + log(z + 1) − γ −2 log z,
1298
D. WANG
then we get:
−1
)z+γ
• f (z) = − ((1+γz(z+1)
−1 )
1
) = 0;
• f (− γ +1
1
• f (− γ +1
)=
2(γ +1)4
γ3
1
, with the zero point z = − γ +1
;
> 0.
1
,
So locally around z = − γ +1
f −
(48)
1
γ +1
+w =
+ log γ − (1 − γ −2 ) log(γ + 1) + γ −2 πi
γ +1
γ2
+
(γ + 1)4 3
w + R1 (w),
3γ 3
where
R1 (w) = O(w4 ),
(49)
After the substitution w = z +
%
e2M(−yz+log(z−1)−γ
C
=
%
exp 2M
M
−2 log z)
1
γ +1 ,
as w → 0.
we get
z
dz
((a + 1)z + a)(z + 1)
γ +1
+ log γ − (1 − γ −2 ) log(γ + 1) + γ −2 πi
γ2
+
w − 1/(γ + 1)
dw
((a + 1)w + (aγ − 1)/(γ + 1))(w + γ /(γ + 1))
×
=−
(γ + 1)4 3
(1 + γ )4/3
1
w
+
R
(w)
−
η w−
1
3
2/3
3γ
γ (2M)
γ +1
γ 2M−1
1
e2M/(1+γ )y
a + 1 (γ + 1)2(M−N)
×
%
M
exp
×
−(1 + γ )4/3
(1 + γ )4
(2M)1/3 ηw +
2Mw3 + 2MR1 (w)
γ
3γ 3
1
−(γ + 1)w + 1
dw,
(γ + 1)/γ w + 1 w + (aγ − 1)/((γ + 1)(a + 1))
where M is a contour around
are defined as (see Figure 3)
1
γ +1 ,
composed of 1M , 2M , 3M and 4M , which
1M
γ
1
= (4 − t)
(2M)−1/3 ,
eπ i/3 0 ≤ t ≤ 4 −
γ +1
(1 + γ )1/3
2M
1
γ
−1/3 −tπ i 1
=
(2M)
e
,
− ≤ t ≤
4/3
(1 + γ )
3
3
1299
QUATERNIONIC WISHART
F IG . 3.
M .
1
γ
(2M)−1/3 ≤ t ≤ 4 ,
e5π i/3 γ +1
(1 + γ )1/3
√
√ γ γ
γ
+ it −2 3
≤t ≤2 3
= 2
.
γ +1
γ +1
γ +1
3M = t
4M
For the asymptotic analysis, we define
(50)
(51)
M
local
= {z ∈ M |(z) ≤ (2M)−10/39 },
∞
<c
= {w ∈ ∞ |(w) < c},
M
M
remote
= (1M ∪ 3M ) \ local
,
∞
∞
≥c
= ∞ \ <c
.
Now, we denote
FaM (η, w) =
(1 + γ )4/3
γ (2M)1/3
(1 + γ )4/3
(1 + γ )4
(2M)1/3 ηw +
× exp −
2Mw3 + 2MR1 (w)
γ
3γ 3
×
1
−(γ + 1)w + 1
,
(γ + 1)/γ w + 1 w + (aγ − 1)/((γ + 1)(a + 1))
and establish several lemmas for the proof. L EMMA 2.
If T is fixed and M is large enough, then for any η > T ,
1
2πi
4M
FaM (η, w) dw <
1 e−η/2
.
3 M 1/40
1300
D. WANG
P ROOF.
By (48) and (47),
(1 + γ )4
2Mw3 + 2MR1 (w)
3γ 3
= 2M f −
1
γ +1
+w −
γ +1
γ2
− log γ + (1 − γ −2 ) log(γ + 1) − γ −2 πi
(52)
γ +1
(γ + 1)2
= 2M −
w + log
w+1
2
γ
γ
−γ
−2
log (γ + 1)w − 1 − γ
−2
πi .
If w ∈ 4M , (w) = 2 γ γ+1 , and denote θ = arg(w) ∈ [− π3 , π3 ], we have
(1 + γ )4
2Mw3 + 2MR1 (w)
3γ 3
= 2M −2
γ +1
+ log 32 + (2 tan θ )2
γ
− γ −2 log
(1 + 2γ )2 + (2γ tan θ )2
√
√
γ +1
+ log 21 − γ −2 log(1 + 2γ ) < log 21 − 2 2M < 0.
γ
√
So on 4M , if η ≥ T , 0 < ε < 2 − log 21 and M large enough,
≤ 2M −2
|FaM (η, w)| <
(1 + γ )4/3
(2M)1/3
γ
× exp −2(η − T )(1 + γ )1/3 (2M)1/3
√
+ (log 21 − 2) − 2T (1 + γ )−2/3 2M
−(γ + 1)w + 1
× < e−2(η−T )(1+γ )
1/3 (2M)1/3
√
e(log 21−2+ε )2M ,
√
where ε is a positive number and ε < 2 − log 21. If M is large enough,
e(log
(53)
1
(γ + 1)/γ w + 1 w + (aγ − 1)/((γ + 1)(a + 1)) √
21−2+ε )2M
e−2(η−T )(1+γ )
1/3 (2M)1/3
2π
1 e−T /2
< √
,
2 3γ /(γ + 1) 3 M 1/40
< eT /2 e−η/2 ,
1301
QUATERNIONIC WISHART
and we get the result, since
√
2 3γ /(γ + 1)
max |FaM (η, w)|.
FaM (η, w) dw ≤
2π
4M
w∈4M
1
2πi
(54)
L EMMA 3.
If T is fixed and M is large enough, then for any η > T ,
1
2πi
1 e−η/2
FaM (η, w) dw <
.
M
3 M 1/40
remote
M
P ROOF. For w ∈ remote
, we denote l = (w) =
get by (52)
|w|
2 .
Since arg(w) = ± π3 , we
(1 + γ )4
2Mw3 + 2MR1 (w)
3γ 3
γ +1
(γ + 1)2
1
γ +1
l+4
l
= 2M −
l + log 1 + 2
γ2
2
γ
γ
2 −
γ −2
log 1 − 2(γ + 1)l + 4(γ + 1)2 l 2 .
2
Then we take derivative
d
γ +1
(γ + 1)2
1
γ +1
l+4
l
−
l + log 1 + 2
dl
γ2
2
γ
γ
2 −
γ −2
log 1 − 2(γ + 1)l + 4(γ + 1)2 l 2
2
γ +1
(γ + 1)4 2
γ +1
l+4
l
= −8
l 1 − (γ − 1)
γ3
γ
γ
×
1+2
γ +1
γ +1
l+4
l
γ
γ
2 2 −1
2 2
× 1 − 2(γ + 1)l + 4(γ + 1) l
,
and are able to find a positive number ε > 0, such that for 0 < l ≤ 2 γ γ+1 ,
−8
(γ + 1)4 2
l
γ3
×
1 − (γ − 1)(γ + 1)/γ l + 4((γ + 1)/γ l)2
(1 + 2(γ + 1)/γ l + 4((γ + 1)/γ l)2 )(1 − 2(γ + 1)l + 4(γ + 1)2 l 2 )
< 3ε l 2 ,
1302
D. WANG
M
and
two left-most points of remote
, (1 +
√ on the−10/39
3i)(2M)
,
√
3i)(2M)−10/39 and (1 −
(1 + γ )4
3
2Mw
+
2MR(w)
√
3γ 3
w=(1± 3i)M −10/39
= 2M −
=−
8 (γ + 1)4
(2M)−10/13 + O(M −40/39 )
3 γ3
8 (γ + 1)4
(2M)3/13 1 + O(M −1/39 )
3
3 γ
(2M)−10/39
< 2M
0
−3ε t 2 dt.
M
,
Therefore we know that for w ∈ remote
l
(1 + γ )4
2Mw3 + 2MR1 (w) < 2M
−3ε t 2 dt = −2Mε l 3 ,
3
3γ
0
and have the estimation that if η ≥ T , 0 < ε < ε and M large enough [l ≥
(2M)−10/39 ],
|FaM (ξ, w)| <
(1 + γ )4/3
(2M)1/3
γ
× exp −(η − T )
(1 + γ )4/3
(2M)1/3 l
γ
(1 + γ )4/3
(2M)−2/3 l 2M
− ε l +T
γ
3
−(γ + 1)w + 1
1
×
(γ + 1)/γ w + 1 w + (aγ − 1)/((γ + 1)(a + 1)) < e−(η−T )(1+γ )
4/3 /γ (2M)1/13 −ε (2M)3/13
.
Now we get the result by inequalities similar to (53)–(54). L EMMA 4.
P ROOF.
If T is fixed and c is large enough,
1
1
−T u+u3 /3
du < .
2πi ∞ e
c
≥c
Obvious. L EMMA 5.
1
2πi
If T is fixed and M is large enough, then for any η > T , holds:
M
local
FaM (η, w) dw −
1 e−η/2
(γ + 1)(a + 1)
Ai(η) <
.
1 − aγ
3 M 1/40
1303
QUATERNIONIC WISHART
M , |w| ≤ 2(2M)10/39 , so by (49)
On local
P ROOF.
FaM (η, w) =
(γ + 1)(a + 1) (1 + γ )4/3
(2M)1/3
aγ − 1
γ
× e−(1+γ )
After the substitution u =
M
local
4/3 /γ (2M)1/3 ηw+(1+γ )4 /(3γ 3 )2Mw 3 (1+γ )4/3
(2M)1/3 w,
γ
FaM (η, w) dw =
(55)
1 + O(M −1/39 ) .
we get
(γ + 1)(a + 1)
(aγ − 1)
e−ηu+u
∞
3 /3
du
<(1+γ )4/3 /γ (2M)1/13
× 1 + O(M −1/39 ) ,
and the O(M −1/39 ) term is independent to w.
On ∞ , if η > T , e−(η−T )u < eT /2 e−η/2 . By (2) and (55), we have
1
2πi
(γ + 1)(a + 1)
FaM (η, w) dw −
Ai(η)
M
1 − aγ
local
(γ + 1)(a + 1)
< eT /2 e−η/2 2πi(1 − aγ )
∞
T /2 −η/2 (γ + 1)(a + 1)
+e e
2πi(1 − aγ )
×
|e−T u+u
3 /3
| du
≥(1+γ )4/3 /γ (2M)1/13
|e
∞
−T u+u3 /3
| du O(M
−1/39 ),
<(1+γ )4/3 /γ (2M)1/13
and we can get the result by direct calculation. C ONCLUSION OF THE PROOF OF (43).
the convergence in L2 norm:
lim (1 + a)2(N−M)−1 a −2N+1
M→∞
Putting Lemmas 2–5 together, we get
(γ + 1)2(M−N)+1/3
(1 − aγ )(2M)1/3
γ 2M
× e−2M/(1+γ )y ϕ2N−1 (y)χT (η) = Ai(η)χT (η).
On the other hand, for η ∈ [T , ∞),
(56)
lim (1 + γ −1 )2(N−M)−1 eM−N y M−N+1/2 e(1−γ )/(1+γ )My = 1
M→∞
and
(57)
(1 + γ
−1 2(N−M)−1 M−N M−N+1/2 −(1−γ )/(1+γ )My
)
e
y
e
η
≤1+O √
.
M
1304
D. WANG
Therefore, in L2 norm,
lim (1 + a)2(N−M)−1 a −2N+1
M→∞
(1 − aγ )(2M)1/3 M−N
e
ψ2N−1 (y)χ(η)
(γ + 1)2/3 γ 2N−1
= Ai(η)χ(η).
Now we conclude the proof of the −1 < a < γ −1 part of Theorem 1. By Stirling’s formula, we get
lim (2M)2(M−N)
(58)
M→∞
(2N − 1)! 2(N−M) 4N−1
γ
= 1,
e
(2M − 1)!
and then by (34), (38) and Proposition 5, we have the convergence in trace norm
(1 − aγ )(2M)1/3
#4b (ξ, η)χ(η)
χ (ξ )S
M→∞
(1 + γ )2/3
lim
(59)
1
= χ (ξ ) Ai(ξ ) Ai(η) + Ai (ξ )
2
∞
Ai(t) dt χ (η),
η
which implies that in trace norm,
#4b (ξ, η)χ(η) = 0.
lim χ (ξ )S
M→∞
Now we get the desired result
#4 (ξ, η)χ(η) = lim χ (ξ )S
#4a (ξ, η)χ(η) = χ (ξ )S
4 (ξ, η)χ(η),
lim χ (ξ )S
M→∞
M→∞
and in the same way
\$ 4 (ξ, η)χ(η) = χ (ξ )SD
4 (ξ, η)χ(η),
lim χ (ξ )SD
M→∞
# 4 (ξ, η)χ(η) = χ (ξ )IS
4 (ξ, η)χ(η).
lim χ (ξ )IS
M→∞
Therefore, in trace norm
lim P#T (ξ, η)χ(η) = χ (ξ )
M→∞
S4 (ξ, η)
4 (ξ, η)
IS
4 (ξ, η)
SD
4 (η, ξ ) χ (η),
S
and the convergence of Fredholm determinant follows.
3.2. Proof of the a = γ −1 part of Theorem 1. When a = γ −1 , the 1 − aγ −1 in
(η).
(59) vanishes, so we need other asymptotic formulas for ψ2N−1 (η) and ψ2N−1
−1
The approach is similar to that in the a < γ case, so we just sketch the proof.
1305
QUATERNIONIC WISHART
P ROPOSITION 6. For fixed γ ≥ 1, a = γ −1 , ε > 0 and any T , we have the
convergences in L2 norm with respect to ξ or η:
lim γ −2N−1 (1 + γ )4/3 (2M)1/3 eM−N eεξ L2N−1
(2(M−N))
M→∞
× (2Mx)x M−N−1/2 e−Mx χ (ξ )
= −eεξ Ai(ξ )χ(ξ ),
lim γ
−2N
M→∞
(60)
2Me
= −e−εη
M−N −εη
e
∞
∞
y
L(2(M−N))
(2Mt)t M−N−1/2 e−Mt dt χ(η)
2N−1
Ai(t) dt χ(η),
η
lim (1 + a)2(N−M)−1 a −2N+1 eM−N γ −2N+1 e−εη ψ2N−1 (y)χ(η)
M→∞
= e−εη s (1) (η)χ(η),
lim (1 + a)2(N−M)−1 a −2N+1 eM−N
M→∞
(γ + 1)4/3 εξ e ψ2N−1 (x)χ(ξ )
γ 2N (2M)−2/3
= eεξ Ai(ξ )χ(ξ ).
S KETCH OF PROOF OF (60). We perform the same algebraic procedure and
¯¯ M ¯¯ M ¯¯ M
use the contour ¯¯ M = ¯¯ M
1 ∪ 2 ∪ 3 ∪ 4 which is slightly different from the
M in the a < γ −1 case (see Figure 4):
γ
ε/2
M
π i/3 −1/3
¯
¯
e
1 = (4 − t)
(2M)
,
0 ≤ t ≤ 4 −
γ +1
(1 + γ )1/3
¯¯ M
2 =
5
γ ε/2
−1/3 tπ i 1
(2M)
e
,
≤t ≤
4/3
(1 + γ )
3
3
γ
ε/2
(2M)−1/3 ≤ t ≤ 4 ,
e5π i/3 γ +1
(1 + γ )1/3
√
√ γ γ
γ
M
¯
¯
+ it −2 3
≤t ≤2 3
4 = 2
γ +1
γ +1
γ +1
¯¯ M
¯¯ ∞
¯¯ ∞
and for asymptotic analysis, we define ¯¯ M
remote , local , <c and ≥c in the same
way as (50)–(51). Then we get
¯¯ M
3 = t
ϕ2N−1 (y) = −(1 + a)2(M−N)+1 a 2N−1
×
γ 2M
e(2M)/(1+γ )y
(γ + 1)2(M−N)+1
1
×
2πi
%
¯¯ M
exp −
(1 + γ )4/3
(2M)1/3 ηw
γ
1306
D. WANG
F IG . 4.
¯¯ M .
(1 + γ )4
+
2Mw3 + 2MR1 (w)
3
3γ
×
If we denote
FM (η, w) = exp −
×
−(γ + 1)w + 1 dw
.
(γ + 1)/γ w + 1 w
(1 + γ )4/3
(1 + γ )4
(2M)1/3 ηw +
2Mw3 + 2MR1 (w)
γ
3γ 3
−(γ + 1)w + 1 1
,
(γ + 1)/γ w + 1 w
then parallel to Lemmas 2–5, we have:
L EMMA 6.
For any T fixed, and M large enough, if η > T , then
−εη 1
1 e−εη/2
e
<
F
(η,
w)
dw
M
3 M 1/40 .
2πi ¯¯ M
4
L EMMA 7.
For any T fixed, and M large enough, if η > T , then
−εη 1
e
2πi ¯¯ M
remote
L EMMA 8.
FM (η, w) dw<
If T is fixed and c is large enough,
1 e−εη/2
.
3 M 1/40
1
−T u+u3 /3 du 1
e
< .
2πi ¯¯ ∞
u c
≥c
1307
QUATERNIONIC WISHART
L EMMA 9.
For any T fixed, and M large enough, if η > T , then
−εη 1
1 e−εη/2
−η/2 (1)
e
<
F
(η,
w)
dw
−
e
s
(η)
M
3 M 1/40 .
2πi ¯¯ M
local
Using Lemmas 6–9, we get the convergence in L2 norm:
lim (1 + a)2(N−M)−1 a −2N+1
M→∞
(γ + 1)2(M−N)+1 −2M/(1+γ )y
e
γ 2M
× e−εη ϕ2N−1 (y)χ(η)
= e−εη s (1) (η)χ(η).
Furthermore, because of the limit result (56) and (57), we get the L2 convergence
lim (1 + a)2(N−M)−1 a −2N+1 γ −2N+1 eM−N e−εη ψ2N−1 (y)χ(η)
M→∞
= e−εη s (1) (η)χ(η).
Now we conclude the proof of the a > γ −1 part of Theorem 1. Using (34), (58)
and Proposition 6 we have the convergence in trace norm
#4b (ξ, η)e−εη χ (η)
lim χ (ξ )eεξ S
M→∞
1
= χ (ξ )eεξ Ai(ξ )s (1) (η) + Ai(ξ )
2
∞
Ai(t) dt e−εη χ (η)
η
1
= χ (ξ )eεξ Ai(ξ )e−εη χ (η),
2
and this together with the conjugated convergence result (discussed in the Appen#4a (ξ, η) in formulas (39) and (40) of Section 3.1 conclude
dix) of S
(61)
#4 (ξ, η)e−εη χ (η) = χ (ξ )eεξ S 4 (ξ, η)e−εη χ (η).
lim χ (ξ )eεξ S
M→∞
In the same way we get
\$ 4 (ξ, η)eεη χ (η) = χ (ξ )eεξ SD4 (ξ, η)eεη χ (η),
lim χ (ξ )eεξ SD
M→∞
# 4 (ξ, η)e−εη χ (η) = χ (ξ )e−εξ IS4 (ξ, η)e−εη χ (η).
lim χ (ξ )e−εξ IS
M→∞
Then we get the convergence in trace norm of a conjugate of P#T (ξ, η)
lim χ (ξ )
M→∞
#4 (ξ, η)e−εη
eεξ S
−εξ
# 4 (ξ, η)e−εη
e IS
eεξ S (ξ, η)e−εη
= χ (ξ ) −εξ 4
e IS4 (ξ, η)e−εη
\$ 4 (ξ, η)eεη
eεξ SD
−εξ
#4 (η, ξ )eεη χ (η)
e S
eεξ SD4 (ξ, η)eεη
χ (η),
e−εξ S 4 (η, ξ )eεη
and the convergence of Fredholm determinant follows.
1308
D. WANG
3.3. Proof of the a > γ −1 part of Theorem 1. If a > γ −1 , the location as well
as the fluctuation scale of the largest sample eigenvalue
is changed. We change
1
variables as pM = (a + 1)(1 + γ 2 a ) and qM = (a + 1) 1 − γ 21a 2 √ 1 , and then by
2M
(36) the kernel S∗ (x, y) after substitution is [here ∗ stands for 4, 4a or 4b, and the
#∗ (ξ, η) in this subsection is not identical to that in Sections 3.1 and 3.2]
S
&
#∗ (ξ, η) = (a + 1) 1 −
S
(62)
1
γ 2a2
1
√
2M
× S∗ (x, y)|x=(a+1)(1+1/(γ 2 a))+(a+1)√1−1/(γ 2 a 2 )1/(√2M)ξ .
√
√
y=(a+1)(1+1/(γ 2 a))+(a+1)
1−1/(γ 2 a 2 )1/( 2M)η
#4b (ξ, η) first.
We analyze S
P ROPOSITION 7. For fixed γ ≥ 1, a > γ −1 , ε > 0 and any T , we have convergences in L2 norm with respect to ξ or η:
(γ 2 a + 1)M−N+1/2
(γ 2 a 2 − 1)2MeM−N
M→∞ (γ 2 a)M+N+1/2 (a + 1)M−N−1/2
lim
× e(γ
(63)
2 a 2 −1)/((γ 2 a+1)(a+1))Mx
× eεξ L(2(M−N))
(2Mx)x M−N−1/2 e−Mx χ (ξ )
2N−1
1
1 γ 4 a 2 + γ 2 a 2 + 4γ 2 a + γ 2 + 1 2
= − √ exp −
ξ + εξ χ (ξ ),
4
(γ 2 a + 1)2
2π
1 (γ 2 a + 1)M−N−1/2 (γ 2 a 2 − 1) 2 2
γ a − 1(2M)3/2 eM−N
M→∞ 2 (γ 2 a)M+N+1/2 (a + 1)M−N+1/2
lim
× e(γ
(64)
×
2 a 2 −1)/((γ 2 a+1)(a+1))My
∞
y
eεη
L(2(M−N))
(2Mt)t M−N−1/2 e−Mt dt χ(η)
2N−1
1
4 2
2 2
2
2
2
2 2
= − √ e−1/4(γ a +γ a +4γ a+γ +1)/(γ a+1) η +εη χ (η),
2π
lim
M→∞
(65)
γ 2a
(γ 2 a + 1)(a + 1)
× e−(γ
M−N+1/2
2 a 2 −1)/((γ 2 a+1)(a+1))My
= e−1/4(γ
eM−N
e−εη ψ2N−1 (y)χ(η)
2 a 2 −1)(γ 2 −1)/(γ 2 a+1)2 η2 −εη
χ (η),
1309
QUATERNIONIC WISHART
lim
M→∞
γ 2a
(γ 2 a + 1)(a + 1)
× e−(γ
(66)
M−N−1/2
2 a 2 −1)/((γ 2 a+1)(a+1))Mx
= e−1/4(γ
eM−N
γ 2a
(γ 2 a 2 − 1)M
e−εξ ψ2N−1
(x)χ(ξ )
2 a 2 −1)(γ 2 −1)/(γ 2 a+1)2 ξ 2 −εξ
χ (ξ ).
We only prove (65). The identity (45) still holds, but we need to use another
contour and a new procedure of steepest-descent analysis.
Since
e−2Myz
(z + 1)2M
z2N
= e2M(−(a+1)(1+1/(γ
2 a))z+log(z+1)−γ −2 log z)−(a+1)
√
√
1−1/(γ 2 a 2 ) 2Mηz
,
if we denote (ignoring the ambiguity of values of logarithm)
g(z) = −(a + 1) 1 +
1
γ 2a
z + log(z + 1) − γ −2 log z,
then we get:
• g (z) = −(a + 1)(1 +
a
z = − 1+a
;
1
γ 2a
)+
1
z+1
−
γ −2
z ,
γ −2
, g (− 1+γ1 2 a )
z2
a
g (− 1+a
) = (1 + a)2 ( γ 21a 2 − 1) < 0.
1
• g (z) = − (z+1)
2 +
with zero points z = − 1+γ1 2 a and
= (γ −1 + γ a)2 (1 −
1
)
γ 2 a2
> 0 and
So we take z = − 1+γ1 2 a as the saddle point, and locally around that point, after the
substitution w = z +
g −
1
,
1+γ 2 a
we get
1
a+1
+ w = 2 + log(γ 2 a) − (1 − γ −2 ) log(γ 2 a + 1) + γ −2 πi
2
1+γ a
γ a
1
1
+ (γ −1 + γ a)2 1 − 2 2 w2 + R2 (w),
2
γ a
where
R2 (w) = O(w3 )
as w → 0,
so that
%
e2M(−yz+log(z+1)−γ
C
(67) =
%
M
exp 2M
−2 log z)
z
dz
((a + 1)z + a)(z + 1)
a+1
+ log(γ 2 a) − (1 − γ −2 ) log(γ 2 a + 1)
γ 2a
1310
D. WANG
1
1
+ γ −2 πi + (γ −1 + γ a)2 1 − 2 2 w2 + R2 (w)
2
γ a
&
η
1
1
w− 2
− (a + 1) 1 − 2 2 √
γ a
γ a+1
2M
×
=−
w − 1/(γ 2 a + 1)
dw
((a + 1)w + (γ 2 a 2 − 1)/(γ 2 a + 1))(w + (γ 2 a)/(γ 2 a + 1)))
1
(γ 2 a)2M−1
2
e2M/(γ a+1)x
a + 1 (γ 2 a + 1)2(M−N)
×
%
M
&
exp −(a + 1) 1 −
1 √
2Mηw
γ 2a2
1
1
+ (γ −1 + γ a)2 1 − 2 2 2Mw2 + 2MR2 (w)
2
γ a
×
×
−(γ 2 a + 1)w + 1
(γ 2 a + 1)/(γ 2 a)w + 1
w + (γ 2 a 2
where M is a contour around
are defined as (see Figure 5)
1
dw,
− 1)/((γ 2 a + 1)(a + 1))
1
, composed of 1M , 2M , 3M
γ 2 a+1
1M = {−it|−2 ≤ t ≤ 2},
3M = {4 + it|−2 ≤ t ≤ 2},
and 4M , which
2M = {4 − t + 2i|0 ≤ t ≤ 4},
4M = {t − 2i|0 ≤ t ≤ 4}.
And for the asymptotic analysis, we define (see Figure 6)
M
= {w ∈ M ||w| ≤ M −2/5 },
local
F IG . 5.
M
M
remote
= 1M \ local
,
M .
1311
QUATERNIONIC WISHART
∞.
F IG . 6.
∞ = {−it|−∞ < t < ∞},
∞
<c
= {w ∈ ∞ ||w| < c},
∞
∞
= ∞ \ <c
.
≥c
Then if we denote
GaM (η, w) = (γ
−1
&
+ γ a)
1
1 − 2 2 2M
γ a
&
× exp −(a + 1)
1−
1
2Mηw
2
γ a2
1
1
+ (γ −1 + γ a)2 1 − 2 2 2Mw2 + 2MR2 (w)
2
γ a
×
1
−(γ 2 a + 1)w + 1
,
2
2
2
2
(γ a + 1)/(γ a)w + 1 w + (γ a − 1)/((γ 2 a + 1)(a + 1))
we have four lemmas similar to Lemmas 2–5:
L EMMA 10.
For any T fixed, and M large enough, if η > T , then
−εη 1
e
2πi
L EMMA 11.
1 e−εη
GaM (η, w) dw<
.
3 M 1/10
2M ∪3M ∪4M
For any T fixed, and M large enough, if η > T , then
−εη 1
e
2πi
M
remote
L EMMA 12.
GaM (η, w) dw<
1 e−εη
.
3 M 1/10
If T is fixed and c is large enough,
1
2πi
∞
≥c
e−(a+1)/(γ
−1 +γ a)T u+u2 /2
1
du< .
c
1312
D. WANG
L EMMA 13.
For any T fixed, and M large enough, if η > T , then
−εη 1
e
2πi
M
local
GaM (η, w) dw
(γ 2 a + 1)(a + 1) −1/2((γ (a+1))/(γ 2 a+1)η)2
√
e
− εη
−
2
2
(γ a − 1) 2π
<
1 e−εη
.
3 M 1/10
Their proofs are the same as those of Lemmas 2–5, and we need the identity
1
2πi
∞
S KETCH
e−(a+1)/(γ
−1 +γ a)ηu+u2 /2
OF PROOF OF
(65).
1
2
2
du = − √ e−1/2(γ (a+1)/(γ a+1)η) .
2π
a
Because the pole z = − a+1
, which is w =
a −1
in the w plane, is not in side of M , so
− (γ 2γa+1)(a+1)
2 2
%
(z + 1)2(M−N)
dz = 0.
z + a/(a + 1)
C
Similar to but subtler than (56) and (57), if we denote (here we have a notation
conflict with the ri defined in Section 2.3, but there should be no confusion)
(68)
rM (η) =
e−2Mxz
γ 2a
(γ 2 a + 1)(a + 1)
× e−(γ
M−N+1/2
eM−N y M−N+1/2
2 −1)a/((γ 2 a+1)(a+1))My−εη
,
we have for η ∈ [T , ∞),
lim rM (η) = e−1/4(γ
2 a 2 −1)(γ 2 −1)/(γ 2 a+1)2 η2 −εη
M→∞
,
and for a large enough positive C, η ∈ [C, ∞) and M1 < M2 , pointwisely
rM1 (η) > rM2 (η) > 0,
so that we can use the dominant convergence theorem to prove that in L2 norm,
lim rM (η)χ(η) = e−1/4(γ
2 a 2 −1)(γ 2 −1)/((γ 2 a+1)2 )η2 −εη
M→∞
Finally, since from (45), (67) and (68),
ψ2N−1 (y) = y M−N+1/2 e(a−1)/(a+1)My
+ (1 + a)2(M−N) a 2N−1
×
1
(γ 2 a)2M
2
2(M−N)+1
(γ a + 1)
(γ 2 a 2 − 1)2M
χ (η).
1313
QUATERNIONIC WISHART
× y M−N+1/2 e(1−γ
= y M−N+1/2 e−((γ
'
× e(γ
2 a)/(1+γ 2 a)My
1
2πi
M
GaM (η, w) dw
2 −1)a)/((γ 2 a+1)(a+1))My
2 a 2 −1)/(γ 2 a+1)(a+1)My
+ (1 + a)2(M−N) a 2N−1
× e−(γ
(γ 2 a)2M
1
(γ 2 a + 1)2(M−N)+1 (γ 2 a 2 − 1)2M
2 a 2 −1)/((γ 2 a+1)(a+1))My
1
2πi
M
(
GaM (η, w) dw ,
we get
γ 2a
(γ 2 a + 1)(a + 1)
× eM−N e−(γ
M−N+1/2
2 a 2 −1)/((γ 2 a+1)(a+1))My
'
e−εη ψ2N−1 (y)χ (η)
= rM (η) 1 + (1 + a)2(M−N) a 2N−1
(γ 2 a)2M
(γ 2 a + 1)2(M−N)+1
(γ 2 a 2 − 1)
2My
exp − 2
×
(γ a + 1)(a + 1)
(γ 2 a 2 − 1)2M
1
1
×
2πi
M
(
GaM (η, w) dw χ (η),
and get the L2 convergence
lim
M→∞
γ 2a
(γ 2 a + 1)(a + 1)
× eM−N e−(γ
M−N+1/2
2 a 2 −1)/((γ 2 a+1)(a+1))My
= lim rM (η)χ (η) = e−1/4(γ
e−εη ψ2N−1 (y)χ (η)
2 a 2 −1)(γ 2 −1)/(γ 2 a+1)2 η2 −εη
M→∞
χ (η),
because for a > γ −1 and η ∈ [T , ∞), we can verify by by elementary but tricky
estimation that
lim (1 + a)2(M−N) a 2N−1
M→∞
e−(γ a −1)/((γ a+1)(a+1))2My
(γ 2 a)2M
=0
(γ 2 a + 1)2(M−N)+1
(γ 2 a 2 − 1)2M
2 2
×
2
1314
D. WANG
uniformly, and by Lemmas 10–13, in L2 norm
lim e−εη
M→∞
=
1
2πi
M
GaM (η, w) dw χ(η)
(γ 2 a + 1)(a + 1) −1/2(γ (a+1)/(γ 2 a+1)η)2 −εη
√
e
χ (η).
(γ 2 a 2 − 1) 2π
For notational simplicity, we denote functions on the left-hand sides of (63)–
(66) by F1 (ξ )χ(ξ ), F2 (η)χ(η), F3 (η)χ(η) and F4 (ξ )χ(ξ ), and denote
cM = (2M)2(M−N)
(2N − 1)! 2(N−M) 4N−1
e
γ
.
(2M − 1)!
By (58), we have
lim cM = 1.
M→∞
Then we get from (34), (62) and (63)–(66)
#4b (ξ, η) = −
S
(69)
cM −(γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(x−y)−ε(ξ −η)
e
F1 (ξ )F3 (η)
2
+ e(γ
2 a 2 −1)/((γ 2 a+1)(a+1))M(x−y)+ε(ξ −η)
If we define
SD4a (x, y) =
N−2
j =0
IS4a (x, y) =
N−2
j =0
1 ψ2j (x)ψ2j
+1 (y) − ψ2j +1 (x)ψ2j (y) ,
rj
1
−ψ2j (x)ψ2j +1 (y) + ψ2j +1 (x)ψ2j (y) ,
rj
and
SD4b (x, y) =
IS4b (x, y) =
1 rN−1
1 rN−1
ψ2N−2
(x)ψ2N−1
(y) − ψ2N−1
(x)ψ2N−2
(y) ,
−ψ2N−2 (x)ψ2N−1 (y) + ψ2N−1 (x)ψ2N−2 (y) ,
like
S4a (x, y) =
N−2
j =0
S4b (x, y) =
1
−ψ2j
(x)ψ2j +1 (y) + ψ2j
+1 (x)ψ2j (y) ,
rj
1 rN−1
F4 (ξ )F2 (η) .
−ψ2N−2
(x)ψ2N−1 (y) + ψ2N−1
(x)ψ2N−2 (y) ,
1315
QUATERNIONIC WISHART
in (28) and (29), and by (35) and (37) define [∗ stands for 4, 4a or 4b]
\$ ∗ (ξ, η) = (a + 1)2 1 −
SD
1
1
γ 2 a 2 2M
× SD∗ (x, y)|x=(a+1)(1+1/(γ 2 a))+(a+1)√1−1/(γ 2 a 2 )1/(√2M)ξ ,
√
√
y=(a+1)(1+1/(γ 2 a))+(a+1)
1−1/(γ 2 a 2 )1/( 2M)η
√
# ∗ (ξ, η) = IS∗ (x, y)|
√
IS
,
x=(a+1)(1+1/(γ 2 a))+(a+1) 1−1/(γ 2 a 2 )1/( 2M)ξ
√
√
y=(a+1)(1+1/(γ 2 a))+(a+1)
1−1/(γ 2 a 2 )1/( 2M)η
like
#∗ (ξ, η) = (a + 1) 1 − 1/(γ 2 a 2 ) √ 1
S
2M
× S∗ (x, y)|x=(a+1)(1+1/(γ 2 a))+(a+1)√1−1/(γ 2 a 2 )1/(√2M)ξ
√
√
y=(a+1)(1+1/(γ 2 a))+(a+1)
1−1/(γ 2 a 2 )1/( 2M)η
in (62), then in the same way of (69), we have
\$ 4b (ξ, η) =
SD
cM
2 2
2
CM e−(γ a −1)/((γ a+1)(a+1))M(x−y)−ε(ξ −η) F1 (ξ )F4 (η)
4
− e(γ
# 4b (ξ, η) =
IS
2 a 2 −1)/((γ 2 a+1)(a+1))M(x−y)+ε(ξ −η)
F4 (ξ )F1 (η) ,
cM −(γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(x−y)−ε(ξ −η)
e
F2 (ξ )F3 (η)
CM
− e(γ
2 a 2 −1)/((γ 2 a+1)(a+1))M(x−y)+ε(ξ −η)
with
CM
√
(γ 2 a 2 − 1)3/2 2M
.
=
aγ (γ 2 a + 1)
Now we write P#T (ξ, η) as the sum
P#T (ξ, η) = P#T a (ξ, η) + P#T b (ξ, η),
with
P#T a (ξ, η) = χ (ξ )
#
S4a (ξ, η)
# 4a (ξ, η)
IS
#
\$ 4a (ξ, η)
SD
#4a (η, ξ ) χ (η),
S
F3 (ξ )F2 (η) ,
\$ 4b (ξ, η)
S (ξ, η) SD
P#T b (ξ, η) = χ (ξ ) #4b
#4b (η, ξ ) χ (η).
IS4b (ξ, η) S
1316
D. WANG
If we denote
⎛
γ 2a2 − 1
M(x − x0 ) + εξ
⎜ exp
U (ξ ) = ⎝
(γ 2 a + 1)(a + 1)
0
CM F4 (ξ ) (γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(x−x0 )+εξ ⎞
e
−
⎟
2 F3 (ξ )
⎠,
⎛
e(γ
2 a 2 −1)/((γ 2 a+1)(a+1))M(x−x )+εξ
0
2 2
2
⎜ e−(γ a −1)/((γ a+1)(a+1))M(y−y0 )−εη
U −1 (η) = ⎝
0
CM F4 (ξ ) −(γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(y−y0 )−εξ ⎞
e
⎟
2 F3 (ξ )
⎠,
e(γ
2 a 2 −1)/((γ 2 a+1)(a+1))M(y−y )+εξ
0
with
x0 = y0 = (a + 1) 1 +
1
&
γ 2a
+ (a + 1) 1 −
1
γ 2a2
T
√
,
2M
then we have the result of kernel conjugation
⎛ c F2 (ξ )F4 (ξ )
M
−
F
(ξ
)
+
F3 (η)
1
F3 (ξ )
U (ξ )P#T b (ξ, η)U −1 (η) = χ (ξ ) ⎝ 2
U (ξ )P#T b (ξ, η)U −1 (η)21
⎞
0
CM
F2 (η)F4 (η) ⎠ χ (η),
−
F3 (ξ ) F1 (η) +
2
F3 (η)
with the entry
U (ξ )P#T b (ξ, η)U −1 (η)21
cM −2(γ 2 a 2 −1)/((γ 2 a+1)(a+1))M(x−x0 )−2εξ
=
e
F2 (ξ )F3 (η)
CM
− F3 (ξ )F2 (η)e−2(γ
2 a 2 −1)/((γ 2 a+1)(a+1))M(y−y )−2εη 0
.
We want U (ξ )P#T b (ξ, η)U −1 (η) to converge in trace norm as M → ∞, and
need the results:
L EMMA 14.
In trace norm,
lim U (ξ )P#T b (ξ, η)U −1 (η)21 = 0.
M→∞
1317
QUATERNIONIC WISHART
L EMMA 15.
In L2 norm,
F2 (ξ )F4 (ξ )
χ (ξ )
M→∞
F3 (ξ )
lim
1
1 γ 4 a 2 + γ 2 a 2 + 4γ 2 a + γ 2 + 1 2
= − √ exp −
η + εξ χ (ξ ).
4
(γ 2 a + 1)2
2π
The proof of Lemma 14 is obvious. The main ingredient in the proof of
Lemma 15 is (64) and the fact that F4 (ξ )/F3 (ξ ) approaches to 1 uniformly on
[T , ∞).
We need another convergence result on U (ξ )P#T a (ξ, η)U −1 (η):
P ROPOSITION 8.
(70)
In trace norm,
lim U (ξ )P#T a (ξ, η)U −1 (η) = 0.
M→∞
The proof is left to the reader. Since all the four entries in P#T a (ξ, η) can be
expressed by Laguerre polynomials like (32) and (33), the asymptotic results like
(63) and (64) give the convergence (70).
By Lemmas 14 and 15 and Proposition 8, we get in trace norm
lim det I − P#T (ξ, η)
M→∞
= lim det I − U (ξ )P#T (ξ, η)U −1 (η)
M→∞
= lim det I − U (ξ )P#T b (ξ, η)U −1 (η)
M→∞
=
T
∞
1
2
√ e−t /2 dt
2π
2
,
and we get the proof of the a > γ −1 part of Theorem 1.
4. Proof of FGSE1 = FGOE . In manipulation of kernels, we follow the method
of [30]. The procedure seems informal and cursory, but is carefully justified in [30].
For notational simplicity, we denote [χ (ξ ) = χ(T ,∞) (ξ )]
B(ξ ) = 1 − s (1) (ξ ) =
∞
Ai(t) dt.
ξ
First, we express the integral operator
χ (ξ )S 4 (ξ, η)χ(η) χ(ξ )SD4 (ξ, η)χ(η)
χ (ξ )P (ξ, η)χ (η) =
χ (ξ )IS4 (ξ, η)χ(η) χ(ξ )S 4 (η, ξ, )χ (η)
1318
D. WANG
by
χ (ξ )
∂
∂ξ
0
0
χ (ξ )
IS4 (ξ, η)χ(η) S 4 (η, ξ )χ(η)
,
IS4 (ξ, η)χ(η) S 4 (η, ξ, )χ (η)
since by (22)–(23) and taking limit,
∂
IS4 (ξ, η) = S 4 (ξ, η),
∂ξ
∂
S 4 (η, ξ ) = SD4 (ξ, η).
∂ξ
Then using (21) for A bounded and B trace class, upon suitably defining the
Hilbert spaces our operators A and B are acting on, we find
∂
0
χ (ξ )
IS4 (ξ, η)χ(η) S 4 (η, ξ )χ(η)
det I −
∂ξ
IS4 (ξ, η)χ(η) S 4 (η, ξ, )χ (η)
0
χ (ξ )
⎛
⎞⎞
⎛
∂
0
χ
(η)
IS
(ξ,
η)χ(η)
S
(η,
ξ
)χ(η)
4
4
⎠⎠
⎝
= det ⎝I −
∂η
IS4 (ξ, η)χ(η) S 4 (η, ξ, )χ (η)
0
χ (η)
⎛
⎛
⎞
⎞
∂
IS (ξ, η)χ(η)
S 4 (η, ξ )χ(η)
⎜
⎜ 4
⎟⎟
∂η
⎜
⎟⎟ ,
= det ⎜
I
−
⎝
⎝
⎠⎠
∂
IS4 (ξ, η)χ(η)
S 4 (η, ξ, )χ (η)
∂η
and by conjugation with
⎛
⎛
1 0
−1 1
, we get
⎞⎞
∂
IS4 (ξ, η)χ(η) + S 4 (η, ξ )χ(η) S 4 (η, ξ )χ(η) ⎠⎠
= det ⎝I − ⎝
∂η
0
0
∂
= det I − IS4 (ξ, η)χ(η) + S 4 (η, ξ )χ(η)
∂η
Since
∞
∂
η=∞
IS4 (ξ, η) f (η) dη = IS4 (ξ, η)f (η)|η=T −
∂η
T
as an operator
IS4 (ξ, η)χ(η)
.
∞
∂
T
∂η
IS4 (ξ, η)f (η) dη,
∂
∂
= IS4 (ξ, ∞)δ∞ (η) − IS4 (ξ, T )δT (η) − IS4 (ξ, η)χ(η),
∂η
∂η
where δ∞ and δT are (generalized) Dirac functions. Then with the help of identity
∞
ξ
KAiry (t, η) dt +
∞
η
KAiry (ξ, t) dt =
∞
ξ
Ai(t) dt
∞
ξ
Ai(t) dt,
1319
QUATERNIONIC WISHART
which can be proved directly from (1), we get
I − IS4 (ξ, η)χ(η)
∂
+ S 4 (η, ξ )χ(η)
∂η
1
= I − KAiry (ξ, η) − B(ξ ) Ai(η) + Ai(η) χ (η)
2
+
∞
1
2
T
1
1
1
KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) δT (η)
4
2
2
1
+ B(ξ )δ∞ (η).
2
Now we denote R(ξ, η) as the resolvent of KAiry (ξ, η)χ(η), such that as integral
operators
−1
I + R(ξ, η) = I − KAiry (ξ, η)χ(η)
(71)
,
then
∂
I − IS4 (ξ, η)χ(η) + S 4 (η, ξ )χ(η)
∂η
= I − KAiry (ξ, η)χ(η)
1
× I − (I + R) 1 − B(ξ ) Ai(η)χ(η)
2
∞
1
+ (I + R)
2
T
KAiry (ξ, t) dt
1
1
1
− B(T )B(ξ ) − B(ξ ) + B(T ) δT (η)
4
2
2
1
+ (I + R)B(ξ )δ∞ (η) .
2
Again by the formula (21), in the form of (formula (17) in [30])
(72)
det I −
n
αk ⊗ βk = det δj,k − (αj , βk )
j,k=1,...,n
k=1
we get
1
det I − (I + R) 1 − B(ξ ) Ai(η)χ(η)
2
∞
1
1
1
1
+ (I + R)
KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) δT (η)
2 T
4
2
2
1320
D. WANG
⎛
1 + α11
= det ⎝ α21
α31
⎞
α12
1 + α22
α32
α13
α23 ⎠ ,
1 + α33
where upon the definition
f (ξ ), g(ξ )T =
we define
)
∞
1
1
+ (I + R)B(ξ )δ∞ (η)
2
∞
f (ξ )g(ξ ) dξ,
T
α11 = (I + R) 1 − 12 B(ξ ) , − Ai(ξ ) T ,
α12 = (I + R)
2
1
1
1
KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) ,
4
2
2
T
*
− Ai(ξ ) ,
α13 =
1
2 (I
T
+ R)B(ξ ), − Ai(ξ ) T ,
α21 = (I + R) 1 − 12 B(ξ ) ξ =T ,
∞
1
α22 = (I + R)
2
T
1
1
1
KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) ,
4
2
2
ξ =T
α23 = 12 (I + R)B(ξ )|ξ =T ,
α31 = (I + R) 1 − 12 B(ξ ) ξ =∞ = 1,
∞
1
α32 = (I + R)
2
T
1
1
1
KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) 4
2
2
ξ =∞
1
= B(T ),
2
α33 = 12 (I + R)B(ξ )|ξ =∞ = 0.
If we take elementary row operations, we get
⎛
1 + α11
det ⎝ α21
α31
⎛
α12
1 + α22
α32
1 + α11 − α13
⎝
= det
α21 − α23
0
= det
1 + β11
β21
⎞
α13
α23 ⎠
1 + α33
α12 − 12 B(T )α13
1 + α22 − 12 B(T )α23
0
β12
,
1 + β22
⎞
α13
α23 ⎠
1
1321
QUATERNIONIC WISHART
where
β11 = (I + R) 1 − B(ξ ) , − Ai(ξ ) T ,
)
∞
1
β12 = (I + R)
2
T
KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) , − Ai(ξ ) ,
T
β21 = (I + R) 1 − B(ξ ) ξ =T ,
∞
1
β22 = (I + R)
2
KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) T
*
ξ =T
.
Using (71) and (72), we observe [s (1) (ξ ) = 1 − B(ξ )]
det I − KAiry (ξ, η)χ(η) det
1 + β11
β21
β12
1 + β22
= det I − KAiry (ξ, η)χ(η) + s (1) (ξ ) Ai(η) χ (η)
1
+
2
∞
T
KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) δT (η) .
If we denote R̃(ξ, η) as the resolvent of (KAiry (ξ, η)χ(η) + s (1) (ξ ) Ai(η))χ(η),
so that as operators
−1
I + R̃(ξ, η) = I + KAiry (ξ, η)χ(η) + s (1) (ξ ) Ai(η) χ (η)
and
Q(ξ ) = (I + R̃)
then
∞
T
KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ) ,
FGSE1 = det I − KAiry (ξ, η)χ(η) + s (1) (ξ ) Ai(η) χ (η) det I + 12 Q(ξ )δT (η) .
To prove Theorem 2, we need only (6) and
det I + 12 Q(ξ )δT (η) = 1,
which by (72) is equivalent to
Q(T ) = 0.
(73)
If we take f (ξ ) = Q(ξ ) + 1, then (73) is
=
∞
T
which is equivalent to
(74)
I − KAiry (ξ, η)χ(η) + s (1) (ξ ) Ai(η) χ (η) f (ξ ) − 1
KAiry (ξ, t) dt − B(T )B(ξ ) − B(ξ ) + B(T ),
I − KAiry (ξ, η)χ(η) + s (1) (ξ ) Ai(η) χ (η) f (ξ ) = s (1) (ξ ).
1322
D. WANG
The integral equation (74) is solvable, and the solution is
f (ξ ) =
(I + R)s (1) (ξ )
.
1 − (I + R)s (1) (ξ ), Ai(ξ )T
Therefore to prove Theorem 2 we need only to prove f (T ) = 1, which is equivalent to
(I + R)s (1) (T ) = 1 − (I + R)s (1) (ξ ), Ai(ξ ) T .
This is a nontrivial result, but it can be derived by results in [30]. In Section VII of
[30] Tracy and Widom√define function q̄ and ū for both GOE and GSE. Our (I +
R)s (1) (T ) is equal to 2 times their q̄ in GOE and our (I + R)s (1) (ξ ), Ai(ξ )T
is equal to 2 times their ū in GOE. With
(I + R)s (1) (T ) = e−
(75)
+∞
T
q(s) ds
(I + R)s (1) (ξ ), Ai(ξ ) T = 1 − e−
(76)
+∞
T
,
q(s) ds
,
where q is the Painlevé II function determined by the differential equation
q (s) = sq(s) + 2q 3 (s)
together with the condition q(s) ∼ Ai(s) as s → ∞.
We can give a proof of (75) and (76), based on the method and results in [29].
First, assume T is fixed, then (I + R)s (1) is a function, and we have
'
(
d
ds (1) (ξ )
d
(I + R)s (1) (ξ ) = (I + R)
+
, (1 + R) s (1) (ξ ).
dξ
dξ
dξ
Since
'
d (1)
dξ s (ξ ) = Ai(ξ )
and we have (2.13) in [29], which is
(
d
, (1 + R) = −(2 + R) Ai(ξ ) · (1 − K t )−1 (Ai(η)χ(η)) + R(η, T ) · ρ(T , η),
dξ
where ρ(x, y) = δ(x − y) + R(x, y) is the distribution kernel of 1 + R, and K t is
the transpose (as an operator) of KAiry (ξ, η)χ(η), we have
d
(I + R)s (1) (ξ ) = (1 + R) Ai(ξ ) − (1 + R) Ai(ξ ) · (I + R)s (1) (ξ ), Ai(ξ ) T
dξ
+ R(ξ, T ) · (1 + R)s (1) (T ).
If we regard T as a parameter, then we have
d
(I + R)s (1) (ξ ; T ) = −R(ξ, T ) · (1 + R)s (1) (T ),
dT
because (2.16) in [29] gives
(77)
1
(1 + R) = R(ξ, T ) · ρ(T , η).
dT
1323
QUATERNIONIC WISHART
Therefore, if we set ξ = T and take the derivative with respect to the parameter T ,
we have
d d
d (1)
(1)
+
(1 + R)s (T ) =
(1 + R)s (T ) dT
dξ dT
ξ =T
= (1 + R) Ai(T ) · 1 − (I + R)s (1) (ξ ), Ai(ξ ) T .
On the other hand, by (77) we have
d (I + R)s (1) (ξ ), Ai(ξ ) T
dT
= −(1 + R)s
(1)
)
*
d
(I + R)s (1) (ξ ), Ai(ξ )
(T ) · Ai(T ) +
dT
= −(1 + R)s (1) (T ) · Ai(T ) +
∞
T
R(ξ, T ) Ai(ξ ) dξ
T
= −(1 + R)s (1) (T ) · (1 + R) Ai(T ).
(1.11) and (1.12) in [29] give the result
(1 + R) Ai(T ) = q(T ),
and now if we denote (I + R)s (1) (T ) = sT and (I + R)s (1) (ξ ), Ai(ξ )T = wT ,
we have
⎧
d
⎪
⎪
sT = q(1 − wT )
⎨
dT
⎪
d
⎪
⎩
(1 − wT ) = qsT .
dT
Now we can get (75) and (76) by boundary conditions.
APPENDIX: DISCUSSION ON THE TRACE NORM CONVERGENCE OF
INTEGRAL OPERATORS RELATED TO LUE
For convenience, we write (44) as
%
e2My
z2(M−N)+j
e−2Myz
dz,
2πi C
(z − 1)j +1
where C is a contour around 1, and we have another integral representation of
Laguerre polynomials
(78)
(2(M−N))
Lj
(2My) =
(2(M−N))
Lj
(2Mx) =
(79)
1
(2(M − N) + j )!
2(M−N)
2(M−N)
j !(2M)
x
2πi
×
where D is a contour around 0.
%
D
e2Mxz
(z − 1)j
z2(M−N)+j +1
dz,
1324
D. WANG
Recall the integral operator K(x, y) [11] for the rescaled LUE with parameters
2N and 2M, and by (78) and (79) we have
K(x, y) =
2N−1
j =0
j!
(2M)2(M−N)+1
(2(M − N) + j )!
× L(2(M−N))
(2Mx)L(2(M−N))
(2My)x M−N y M−N e−x+y
j
j
(80)
=
%
2(M−N)+j
%
2M y M−N eMy 2N−1
−2Myz z
dz
dw
e
(2πi)2 x M−N eMx j =0 C
(z − 1)j +1
D
× e2Mxw
(w − 1)j
w2(M−N)+j +1
.
We can write the sum of integrands in (80) as
2N−1
e2Mxz
j =0
z2(M−N)+j −2Myw (w − 1)j
e
(z − 1)j +1
w2(M−N)+j +1
j
= e2Mxz e−2Myw
2N−1
z(w − 1)
z2(M−N)
1
w2(M−N) (z − 1)w j =0 (z − 1)w
= e2Mxz e−2Myw
1 − (z(w − 1)/((z − 1)w))2N
z2(M−N)
1
w2(M−N) (z − 1)w 1 − z(w − 1)/((z − 1)w)
=
1
1
e2Mxz z2(M−N) e−2Myw 2(M−N)
z−w
w
2N
z2M
1
−Myw (w − 1)
e2Mxz
e
.
z−w
(z − 1)2N
w2M
By the residue theorem, let C and D be disjoint, then for the variable z, the pole
z = w is outside of C,
%
%
1
1
e2Mxz z2(M−N) e−2Myw 2(M−N) = 0.
dz
dw
z−w
w
C
D
On the other side, we assume (w − z) to be less than 0, and get
−
1
= 2M
z−w
so that we have
2M
(2πi)2
(81)
=
%
%
dz
dw
C
D
%
2
(2M)
(2πi)2
C
∞
et2M(w−z) dt,
0
2N
z2M
1
2Mxw (w − 1)
e−2Myz
e
z−w
(z − 1)2N
w2M
%
dz
dw
D
∞
0
e−2M(y+t)z
z2M
(z − 1)2N
1325
QUATERNIONIC WISHART
× e2M(x+t)w
= (2M)
2
∞
%
1
2πi
0
×
e
−2M(y+t)z
C
%
1
2πi
z2M
dz
(z − 1)2N
e2M(x+t)w
D
(w − 1)2N
w2M
(w − 1)2N
dw dt.
w2M
Put (80)–(81) together, we get the result
K(x, y) = −(2M)2
×
(82)
y M−N eMy
x M−N eMx
∞
%
1
2πi
0
D
1
×
2πi
e2M(x+t)w
%
e
(w − 1)2N
dw
w2M
−2M(y+t)z
C
z2M
dz dt.
(z − 1)2N
To find the probability that the largest eigenvalue ≥ T in the LUE, we need
to consider the integral operator from L2 ([0, ∞)) to L2 ([0, ∞)) with the kernel
χ (x)K(x, y)χ (y). We can decompose it into the product of two integral operators
by (82):
χ (x)K(x, y)χ (y) = −(2M)2 χ (x)J (x, t)χ[0,∞) (t) ◦ χ[0,∞) (t)H (t, y)χ(y),
where χ (x)J (x, t)χ[0,∞) (t) and χ[0,∞) (t)H (t, y)χ(y) stands for two integral operators with these kernels, and
J (x, t) =
1
x M−N eMx
H (t, y) = y M−N eMy
1
2πi
1
2πi
%
%
e2M(x+t)w
(w − 1)2N
dw,
w2M
e−2M(y+t)z
z2M
dz.
(z − 1)2N
D
C
Since we consider the limiting distribution of the largest eigenvalue around (1 +
)4/3
γ −1 )2 , we take p = (1 + γ −1 )2 , q = γ(1+γ
, x = p + qξ , y = p + qη and t = qτ .
(2M)2/3
#
Then for the rescaled kernel χ (ξ )K(ξ,
η)χ (η), we have
#
#(τ, η)χ (η),
η)χ (η) = χ (ξ )J#(ξ, τ )χ[0,∞) (τ ) ◦ χ[0,∞) (τ )H
χ (ξ )K(ξ,
where
J#(ξ, τ ) =
#(τ, η) =
H
(γ + 1)4/3 1/3 γ 2N eN−M 1
M
γ
x M−N eMx 2πi
(γ
+ 1)4/3
γ
M 1/3
y M−N eMy
γ 2N eN−M
1
2πi
%
e2M(p+q(ξ +τ ))w
D
%
C
(w − 1)2N
dw,
w2M
e−2M(p+q(ξ +τ ))z
z2M
dz.
(z − 1)2N
1326
D. WANG
We want to prove the trace norm convergence
#
η)χ (η)
lim χ (ξ )K(ξ,
M→∞
(83)
= χ (ξ )KAiry (ξ, η)χ(η)
= χ (ξ ) Ai(ξ + τ )χ[0,∞) (τ ) ◦ χ[0,∞) (τ ) Ai(τ + η)χ (η).
By results in functional analysis, we need only to prove the convergence in Hilbert–
Schmidt norm of (e.g., see [20])
(84)
(85)
lim χ (ξ )J#(ξ, τ )χ[0,∞) (τ ) = χ (ξ )Kf Airy (ξ, η)χ(η),
M→∞
#(τ, η)χ (η) = χ[0,∞) (τ ) Ai(τ + η)χ (η).
lim χ[0,∞) (τ )H
M→∞
Since for integral operators, the convergence in Hilbert–Schmidt norm is equivalent to the convergence in L2 norm of their kernels as two variable functions, we
can verify (84) and (85) by asymptotic analysis similar to that in Section 3.
#4a1 (ξ, η)χ(η) in (39), we have
For the integral operator χ (ξ )S
√ #
#(τ, η)χ (η),
#4a1 (ξ, η)χ(η) = 1 χ (ξ ) √1 #
χ (ξ )S
J#(ξ, τ )χ[0,∞) (τ ) ◦ χ[0,∞) (τ ) y H
2
x
#
# in the same way as J# and J#, but use parameters 2N − 2
J# and H
where we define #
#4a1 part of (61), we have
and 2M − 2 instead of 2N and 2M. Similarly, in the S
#4a1 (ξ, η)e−εη χ (η)
χ (ξ )eεξ S
√
y #
eεξ #
1
#(τ, η)χ (η).
= χ (ξ ) √ #
J (ξ, τ )χ[0,∞) (τ ) ◦ χ[0,∞) (τ ) −εη H
2
e
x
We can give rigorous proofs to (39) and (61) in the same way as (83).
who pointed out the spiked model problem and gave warm encouragement to the
author. I also thank Ira Gessel for help in combinatorics, especially his suggestions
in proofs in Section 2, and Jinho Baik for his suggestions on related models.
REFERENCES
[1] A DLER , M., F ORRESTER , P. J., NAGAO , T. and VAN M OERBEKE , P. (2000). Classical skew
orthogonal polynomials and random matrices. J. Statist. Phys. 99 141–170. MR1762659
[2] A NDERSON , T. W. (1963). Asymptotic theory for principal component analysis. Ann. Math.
Statist. 34 122–148. MR0145620
[3] A NDERSSON , S. (1975). Invariant normal models. Ann. Statist. 3 132–154. MR0362703
[4] BAIK , J., B EN A ROUS , G. and P ÉCHÉ , S. (2005). Phase transition of the largest eigenvalue for
nonnull complex sample covariance matrices. Ann. Probab. 33 1643–1697. MR2165575
[5] BAIK , J., B ORODIN , A., R AINS , E. M. and S UIDAN , T. M. (2007). Unpublished results. In
Conference Random and Integrable Models in Mathematics and Physics, September 11–
15, Brussels.
QUATERNIONIC WISHART
1327
[6] BAIK , J. and R AINS , E. M. (2001). The asymptotics of monotone subsequences of involutions.
Duke Math. J. 109 205–281. MR1845180
[7] BAIK , J. and R AINS , E. M. (2001). Symmetrized random permutations. In Random Matrix
Models and Their Applications. Mathematical Sciences Research Institute Publications
40 1–19. Cambridge Univ. Press, Cambridge. MR1842780
[8] BAIK , J. and S ILVERSTEIN , J. W. (2006). Eigenvalues of large sample covariance matrices of
spiked population models. J. Multivariate Anal. 97 1382–1408. MR2279680
[9] DE B RUIJN , N. G. (1955). On some multiple integrals involving determinants. J. Indian Math.
Soc. (N.S.) 19 133–151 (1956). MR0079647
[10] D UMITRIU , I., E DELMAN , A. and S HUMAN , G. (2007). MOPS: Multivariate orthogonal polynomials (symbolically). J. Symbolic Comput. 42 587–620. MR2325917
[11] F ORRESTER , P. J. (1993). The spectrum edge of random matrix ensembles. Nuclear Phys. B
402 709–728. MR1236195
[12] F ORRESTER , P. J. (2000). Painlevé transcendent evaluation of the scaled distribution of the
smallest eigenvalue in the Laguerre orthogonal and symplectic ensembles. Available at
http://www.citebase.org/abstract?id=oai:arXiv.org:nlin/0005064.
[13] F ORRESTER , P. J. (2006). Hard and soft edge spacing distributions for random matrix ensembles with orthogonal and symplectic symmetry. Nonlinearity 19 2989–3002. MR2275509
[14] F ORRESTER , P. J., NAGAO , T. and H ONNER , G. (1999). Correlations for the orthogonalunitary and symplectic-unitary transitions at the hard and soft edges. Nuclear Phys. B 553
601–643. MR1707162
[15] G EMAN , S. (1980). A limit theorem for the norm of random matrices. Ann. Probab. 8 252–261.
MR566592
[16] G OHBERG , I. C. and K RE ĬN , M. G. (1969). Introduction to the Theory of Linear Nonselfadjoint Operators. Translations of Mathematical Monographs 18. Amer. Math. Soc., Providence, RI. Translated from the Russian by A. Feinstein. MR0246142
[17] H ANLON , P. J., S TANLEY, R. P. and S TEMBRIDGE , J. R. (1992). Some combinatorial aspects
of the spectra of normally distributed random matrices. In Hypergeometric Functions on
Domains of Positivity, Jack Polynomials, and Applications (Tampa, FL, 1991). Contemporary Mathematics 138 151–174. Amer. Math. Soc., Providence, RI. MR1199126
[18] J OHANSSON , K. (2001). Discrete orthogonal polynomial ensembles and the Plancherel measure. Ann. of Math. (2) 153 259–296. MR1826414
[19] J OHNSTONE , I. M. (2001). On the distribution of the largest eigenvalue in principal components analysis. Ann. Statist. 29 295–327. MR1863961
[20] L AX , P. D. (2002). Functional Analysis. Pure and Applied Mathematics (New York). Wiley,
New York. MR1892228
[21] M ACDONALD , I. G. (1995). Symmetric Functions and Hall Polynomials, 2nd ed. Oxford Univ.
Press, New York. MR1354144
[22] M AR ČENKO , V. A. and PASTUR , L. A. (1967). Distribution of eigenvalues in certain sets of
random matrices. Mat. Sb. (N.S.) 72 507–536. MR0208649
[23] M EHTA , M. L. (2004). Random Matrices, 3rd ed. Pure and Applied Mathematics (Amsterdam)
[24] M UIRHEAD , R. J. (1982). Aspects of Multivariate Statistical Theory. Wiley, New York.
MR652932
[25] PAUL , D. (2007). Asymptotics of sample eigenstructure for a large dimensional spiked covariance model. Statist. Sinica 17 1617–1642. MR2399865
[26] P RÄHOFER , M. and S POHN , H. (2002). Current fluctuations for the totally asymmetric simple
exclusion process. In In and Out of Equilibrium (Mambucaba, 2000). Progress in Probability 51 185–204. Birkhäuser, Boston. MR1901953
[27] S TANLEY, R. P. (1989). Some combinatorial properties of Jack symmetric functions. Adv.
Math. 77 76–115. MR1014073
1328
D. WANG
[28] S ZEG Ő , G. (1975). Orthogonal Polynomials, 4th ed. Colloquium Publications XXIII. Amer.
Math. Soc., Providence, RI. MR0372517
[29] T RACY, C. A. and W IDOM , H. (1994). Level-spacing distributions and the Airy kernel. Comm.
Math. Phys. 159 151–174. MR1257246
[30] T RACY, C. A. and W IDOM , H. (1996). On orthogonal and symplectic matrix ensembles.
Comm. Math. Phys. 177 727–754. MR1385083
[31] T RACY, C. A. and W IDOM , H. (1998). Correlation functions, cluster functions, and spacing
distributions for random matrices. J. Statist. Phys. 92 809–835. MR1657844
D EPARTMENT OF M ATHEMATICS
B RANDEIS U NIVERSITY
WALTHAM , M ASSACHUSETTS 02454
USA
E- MAIL : [email protected]
```