Given,
A=[12−1−1],B=[ab1−1]
We are given:
(A+B)2=A2+B2
Expanding the left-hand side:
(A+B)2=A2+AB+BA+B2
Therefore,
AB+BA=0
Now,
AB=[a−b2a−b23]
and
BA=[a+2b−2−a−1−b+1]
Thus,
AB+BA=[2a−b+22a−21−a4−b]
Since AB+BA=0,
1−a=0
∴a=1
Also,
4−b=0
∴b=4
Therefore,
a=1,b=4