During the Easter Holidays Madrid experiences a short increase of visitors, but after that, during the months of May and June, you can fully enjoy the city without the crowds. The weather is usually perfect, being both sunny and not too hot. It's a great time for hiking and nature watching in Madrid region and also lines at the greatest attractions of the capital such as Prado Museum are much shorter than usual.
September is also one of the best periods to visit Madrid. After the start of school, you will meet significantly fewer tourists in the centre, and the weather cools down a bit so it's not unbearably hot anymore. Summer-like weather will stay until late October so there is a plenty of time to explore the city and vicinity.