Police said in a statement on Tuesday that Noor Mohammad Tantray, the head of the Jaish-e-Mohammad (JeM) group, and his comrades were shot dead outside the main city of Srinagar on Monday evening.
Indian troops cordoned off a group of houses in Samboora, "which resulted in a fierce gun battle leading to elimination of top JeM commander Noor Mohammad Tantray," the statement said.
Police described the operation as a "significant breakthrough" after Tantray's body was found under debris on Tuesday.
The 47-year-old JeM commander was described by officials as a new "security headache" when he took over as the head of the group.
According to police, Tantray spent 12 years in jail after being convicted in a case of militancy in 2003, but jumped parole two years later to rejoin JeM.
As the news of Tantray's death spread, hundreds of Kashmiri people rushed to the scene, chanting anti-India slogans and hurling stones at the troops.
Security forces used tear gas and pellet guns to disperse angry protesters. At least six protesters were injured.
Since 1989 some armed groups including JeM have been fighting Indian troops and police deployed in the Himalayan territory.
At least 210 anti-India fighters have been killed this year amid a military campaign to rout armed groups resisting the Indian rule in the disputed region. According to police officials and watchdogs, over 80 security personnel and about 60 civilians have also lost their lives in violence across the violence-hit valley.
More than half a million Indian soldiers have been deployed in Kashmir since 1989, when popular calls grew for independence or a merger of the territory with Muslim-majority Pakistan.
A former British colony, Kashmir was divided between India and Pakistan in 1947.
Tens of thousands of people, most of them civilians, have been killed in decades of fighting in Kashmir while the neighbors continue to blame each other for the protracted violence.