Connotations can say a lot more than definitions, and "ministry" has a strong negative connotation.
Words are arbitrary labels which humans assign to concepts. The reason that English-speaking people call an "orange" an "orange" and not a "tomato" is a basically because humans chose to, in theory the labels could be changed. A dictionary is a good way to get a sense of the concept to which the label is assigned.
The dictionary definition of "ministry" is "(in certain countries) a government department headed by a minister of state.' google. and a "minister" is defined as a "head of a government department" google..
Therefore, the direct dictionary definition of the word makes perfect sense. But there is a connotation that is possibly stronger than the definition!! I cringed when I saw the government departments were titled "ministries." It conjures up thoughts of terrible, oppressive governments. I think people make first impressions rapidly, and not always rationally. Perhaps a more neutral moniker should be considered?