Healthy soil microbiomes are crucial for achieving high productivity in combination with crop quality, but our understanding of microbial diversity is still limited. In a large-scale study including 116 composite samples from vineyards, orchards and other crops from all over Styria (south-east Austria), agricultural management as well as distinct soil parameters were identified as drivers of the indigenous microbial communities in agricultural soils. The analysis of the soil microbiota based on microbial profiling of prokaryotic 16S rRNA gene fragments and fungal ITS regions revealed high bacterial and fungal diversity within Styrian agricultural soils; 206,596 prokaryotic and 53,710 fungal OTUs. Vineyards revealed a significantly higher diversity and distinct composition of soil fungi over orchards and other agricultural soils, whereas the prokaryotic diversity was unaffected. Soil pH was identified as one of the most important edaphic modulators of microbial community structure in both, vineyard and orchard soils. In general, the acid-base balance, disorders in the soil sorption complex, content and quality of organic substance as well as individual nutrients were identified as important drivers of the microbial community structure of Styrian vineyard and orchard soils. However, responses to distinct parameters differed in orchards and vineyards, and prokaryotic and fungal community responded differently to the same abiotic factor. In comparison to orchards, the microbiome of vineyard soils maintained a higher stability when herbicides were applied. Orchard soils exhibited drastic shifts within community composition; herbicides seem to have a substantial impact on the bacterial order Chthoniobacterales as well as potential plant growth promoters and antagonists of phytopathogens (Flavobacterium, Monographella), with a decreased abundance in herbicide-treated soils. Moreover, soils of herbicide-treated orchards revealed a significantly higher presence of potential apple pathogenic fungi (Nectria, Thelonectria). These findings provide the basis to adapt soil management practices in the future in order to maintain a healthy microbiome in agricultural soils.