Let the three terms be a,ar,ar2
According to the first condition
a+ar+ar2=26
a(1+r+r2)=26 ... (1)
also (a)2+(ar)2+(ar2)2=364
a2(1+r2+r4)=364 ... (2)
Now [eq.(1)]2eq.(2) we have
[a(1+r+r2)]2a2(1+r2+r4)=(26)2364
(1+r+r2)21+r2+r4=137
13(1+r2+r4)=7(1+r+r2)2
13(1+r2+r4)−7(r2+r+1)2=0
13(r2+r+1)(r2−r+1)−7(r2+r+1)2=0
Using 1+r2+r4=(r2+r+1)(r2−r+1)
13(r2−r+1)−7(r2+r+1)=0
{Dividing by (r2+r+1)}
[13r2−13r+13−7r2−7r−7]=0
(6r2−20r+6)=0
3r2−10r+3=0
(3r−1)(r−3)=0
r=31, r=3
If r=31; a(1+31+321)=26
a9(9+3+1)=26
a=18
Hence the three terms a,ar,ar2 are 18, 6, 2.
If r=3 from eq. (1) we have
a(1+3+32)=26
a(1+3+9)=26
a(13)=26
a=2
So, three terms are 2, 6, 18.