(I) 求常数 $a$ 的值;
（II) 求所作可逆线性变换的矩阵 $\boldsymbol{P}$.
【答案】 【解】(I) 记二次型 $f, g$ 的矩阵分别为 $\boldsymbol{A}, \boldsymbol{B}$, 依题意,
$$\boldsymbol{A}=\left(\begin{array}{ccc} 1-a & 1+a & 0 \\ 1+a & 1-a & 0 \\ 0 & 0 & -2 \end{array}\right), \boldsymbol{B}=\left(\begin{array}{ccc} 1 & 1 & -2 \\ 1 & -2 & 1 \\ -2 & 1 & 1 \end{array}\right), \boldsymbol{A} \text { 与 } \boldsymbol{B} \text { 合同,故 } r(\boldsymbol{A})=r(\boldsymbol{B}) .$$

\begin{aligned} \boldsymbol{A} &=\left(\begin{array}{ccc} 1-a & 1+a & 0 \\ 1+a & 1-a & 0 \\ 0 & 0 & -2 \end{array}\right) \rightarrow\left(\begin{array}{ccc} 1-a & 1+a & 0 \\ 2 & 2 & 0 \\ 0 & 0 & -2 \end{array}\right) \rightarrow\left(\begin{array}{ccc} 1 & 1 & 0 \\ 0 & 2 a & 0 \\ 0 & 0 & 1 \end{array}\right), \\ \boldsymbol{B} &=\left(\begin{array}{ccc} 1 & 1 & -2 \\ 1 & -2 & 1 \\ -2 & 1 & 1 \end{array}\right) \rightarrow\left(\begin{array}{ccc} 1 & 1 & -2 \\ 0 & -3 & 3 \\ 0 & 3 & -3 \end{array}\right) \rightarrow\left(\begin{array}{ccc} 1 & 1 & -2 \\ 0 & -3 & 3 \\ 0 & 0 & 0 \end{array}\right), \end{aligned}

(II) 当 $a=0$ 时, $f\left(x_1, x_2, x_3\right)=x_1^2+x_2^2-2 x_3^2+2 x_1 x_2=\left(x_1+x_2\right)^2-2 x_3^2$, 令 $\left\{\begin{array}{l}z_1=x_1+x_2 \\ z_2=\sqrt{2} x_3 \\ z_3=x_1\end{array}\right.$, 即 $\left\{\begin{array}{l}x_1=z_3 \\ x_2=z_1-z_3 \\ x_3=\frac{1}{\sqrt{2}} z_2\end{array}\right.$, 令 $\boldsymbol{P}_1=\left(\begin{array}{ccc}0 & 0 & 1 \\ 1 & 0 & -1 \\ 0 & \frac{1}{\sqrt{2}} & 0\end{array}\right)$,

\begin{aligned} &g\left(y_1, y_2, y_3\right)=y_1^2-2 y_2^2+y_3^2+2 y_1 y_2-4 y_1 y_3+2 y_2 y_3 \\ &=y_1^2+2 y_1\left(y_2-2 y_3\right)-2 y_2^2+y_3^2+2 y_2 y_3 \\ &=y_1^2+2 y_1\left(y_2-2 y_3\right)+\left(y_2-2 y_3\right)^2-\left(y_2-2 y_3\right)^2-2 y_2^2+y_3^2+2 y_2 y_3 \\ &=\left(y_1+y_2-2 y_3\right)^2-3 y_2^2+6 y_2 y_3-3 y_3^2=\left(y_1+y_2-2 y_3\right)^2-3\left(y_2-y_3\right)^2 \text {, } \\ & \end{aligned}
$$\text { 令 } \boldsymbol{P}=\boldsymbol{P}_1 \boldsymbol{P}_2=\left(\begin{array}{ccc} 0 & 0 & 1 \\ 1 & 0 & -1 \\ 0 & \frac{1}{\sqrt{2}} & 0 \end{array}\right)\left(\begin{array}{ccc} 1 & 1 & -2 \\ 0 & \sqrt{3} & -\sqrt{3} \\ 0 & 0 & 1 \end{array}\right)=\left(\begin{array}{ccc} 0 & 0 & 1 \\ 1 & 1 & -3 \\ 0 & \frac{\sqrt{6}}{2} & -\frac{\sqrt{6}}{2} \end{array}\right) \text {, 贝 }$$
, 则 $P$ 即为所求的可逆矩阵,