Step 1 :Define the function \(f(x, y)=x^{4}-20 x y+2 y^{2}-12\).
Step 2 :Find the first partial derivatives of the function with respect to x and y, \(f_x = 4x^3 - 20y\) and \(f_y = -20x + 4y\).
Step 3 :Set these equal to zero and solve for x and y to find the critical points. The critical points are \((-5, -25), (0, 0), (5, 25)\).
Step 4 :Calculate the second partial derivatives of the function, \(f_{xx} = 12x^2\), \(f_{yy} = 4\), and \(f_{xy} = -20\).
Step 5 :Calculate the determinant of the Hessian matrix, \(D = f_{xx}f_{yy} - (f_{xy})^2 = 48x^2 - 400\).
Step 6 :Evaluate D at the critical points. The values are \(800, -400, 800\).
Step 7 :Apply the second derivative test. If \(D > 0\) and \(f_{xx} > 0\), the function has a relative minimum. If \(D > 0\) and \(f_{xx} < 0\), the function has a relative maximum. If \(D < 0\), the function has a saddle point.
Step 8 :From the above, we see that \(D\) is positive at the points \((-5, -25)\) and \((5, 25)\), and negative at the point \((0, 0)\). Furthermore, \(f_{xx}\) is positive at all these points. Therefore, the function has a relative minimum at the points \((-5, -25)\) and \((5, 25)\), and a saddle point at the point \((0, 0)\).
Step 9 :Final Answer: The function \(f(x, y)=x^{4}-20 x y+2 y^{2}-12\) has a relative minimum at the points \((-5, -25)\) and \((5, 25)\), and a saddle point at the point \((0, 0)\). Therefore, the possible points where \(f(x, y)\) has a relative maximum or minimum are \((-5, -25)\) and \((5, 25)\). So, the final answer is \(\boxed{(-5, -25), (5, 25)}\).