Soil microbial communities play a crucial role in soil fertility, sustainability, and plant health. However, intensive agriculture with increasing chemical inputs and changing environments have influenced native soil microbial communities. Approaches have been developed to study the structure, diversity, and activity of soil microbes to better understand the biology and plant-microbe interactions in soils. Unfortunately, a good understanding of soil microbial community remains a challenge due to the complexity of community composition, interactions of the soil environment, and limitations of technologies, especially related to the functionality of some taxa rarely detected using conventional techniques. Culture-based methods have been shown unable and sometimes are biased for assessing soil microbial communities. To gain further knowledge, culture-independent methods relying on direct analysis of nucleic acids, proteins, and lipids are worth exploring. In recent years, metagenomics, metaproteomics, metatranscriptomics, and proteogenomics have been increasingly used in studying microbial ecology. In this review, we examined the importance of microbial community to soil quality, the mystery of rhizosphere and plant-microbe interactions, and the biodiversity and multi-trophic interactions that influence the soil structure and functionality. The impact of the cropping system and climate change on the soil microbial community was also explored. Importantly, progresses in molecular biology, especially in the development of high-throughput biotechnological tools, were extensively assessed for potential uses to decipher the diversity and dynamics of soil microbial communities, with the highlighted advantages/limitations.