If you are lactose intolerant then you are missing the enzyme the body requires to digest dairy; which is lactase. The ability to digest lactose during the period of breast-feeding is essential to the health of the baby.
However, following the first few months of life, lactase activity starts to decrease. In most humans, this activity declines following weaning to undetectable levels. Meaning many adults are lactose intolerant yet continue to consume dairy regularly, potentially leading to inflammation throughout the body.
Even those of us who are not lactose intolerant are often times sensitive to the protein component of dairy, casein. Molecularly speaking, casein and gluten are very similar. People who are sensitive to one, often cannot tolerate the other either.
Milk protein causes inflammation at a cellular level which impacts mitochondrial function. Your mitochondria determine how your body reacts to the world around you. The better your mitochondria are at creating energy, the better your body and mind will perform, meaning you can do more, and you will feel better doing it. For more information on mitochondria read the book "Head Strong" by Dave Asprey.
Not only is dairy inflammatory but it is also addictive. In some cases when casein breaks down a part of it binds to opiate receptors in the brain and makes you feel sedated. The desire for cheese can be blocked by the same medicines used to reverse opiate drug overdoses in emergency rooms! So if cutting dairy feels impossible try weening yourself off slowly like you would with any other drug.