A body is said to be in equilibrium if there is no net force acting on the body and no net torque acting on the body.
The equilibrium is said to be stable if a small displacement of the body from equilibrium leads to a force that pushes the body back towards its equilibrium position. Equivalently, it is stable if a small displacement increases the potential energy of the body.
The equilibrium is said to be unstable if a small displacement of the body from equilibrium leads to a force that pushes the body away from its equilibrium position. Equivalently, it is unstable if a small displacement decreases the potential energy of the body.
The equilibrium is said to be neutral if a small displacement of the body from equilibrium leaves the body in equilibrium. Equivalently, it is stable if a small displacement does not alter the potential energy of the body.
To find the forces acting on a body when it is in equilibrium we apply three conditions.
We resolve the forces in one direction (usually up and down) and set the NET force to zero.
We resolve the forces in a perpendicular direction (usually sideways) and set the NET force to zero.
We compute the net torque about ANY 1 point in the system and set the net torque to zero. It is usually best to choose as the point to compute the torque a point through which several forces pass or at which several forces act.
A ball at the top of a hill is in unstable equilibrium.
A ball at the bottom of a valley is in stable equilibrium.
A ball on a flat surface is in neutral equilibrium.