The disastrous incident in which Israel Defense Forces (IDF) troops shot dead three Israeli hostages in Gaza on Friday, mistaking them for Hamas fighters despite their waving a white cloth, is a graphic illustration of the risks involved in armed hostage rescue.

Maj Gen Charlie Herbert, a retired British Army general who served on 13 operational tours including Iraq and Afghanistan, told the BBC: "As desperately sad as it is, the shooting dead of the three hostages... again calls into question IDF tactics, proportionality and distinction. One can only imagine how many civilians have been killed in similar circumstances."

The IDF says it takes great care to avoid harming civilians, but the horrific death toll in Gaza - more than 18,000 and rising - has caused Israel to be accused of indiscriminate bombing.

In nearly every case of kidnap in history, those abducted stand a far better chance of emerging alive and unscathed through mediation and a deal, rather than by armed intervention.

One of the first hostage cases I covered was when Al-Qaeda-linked jihadists in Yemen abducted 16 Western tourists in 1998. The British ambassador went to see the interior minister to press for a negotiated release of the hostages. Too late, he was told, the Yemeni army has already gone in. A quarter of those kidnapped were killed in the ensuing firefight, with others injured.

Western and Israeli special forces have spent decades perfecting the art of hostage rescue, but even then it doesn't always go to plan. Israel's Operation Thunderbolt in Entebbe, Uganda in 1976 rescued 102 of the 106 hostages, but the commander of the Israeli commandos was killed. Today his brother, Benjamin Netanyahu, is prime minister.

The 1980 SAS siege of the Iranian embassy in London is perhaps the most iconic hostage rescue of modern times, as it took place in the full glare of the TV cameras.

When a British journalist, Stephen Farrell, was kidnapped by the Taliban in Afghanistan in 2009 he was rescued by British Special Forces. But the raid also resulted in the deaths of one of his rescuers, two civilians and his Afghan interpreter.

The following year a US Navy SEAL team attempted to rescue the British aid worker Linda Norgrove, also kidnapped by the Taliban in Afghanistan. The raid killed her kidnappers but she died from a grenade thrown by one of her rescuers.

In 2012 a British Special Forces operation to free one British and one Italian hostage in Nigeria was ruined when one of the Nigerian soldiers fired his weapon prematurely, alerting the kidnappers who then murdered their captives.

Many hostage cases in Yemen have been resolved peacefully over the years, usually through lengthy tribal negotiations accompanied by interminable narcotic qat-chewing sessions. But ultimately, much depends on the demands and intentions of the kidnappers.

In the case of Al-Qaeda in Iraq and the Islamic State group (IS), the jihadist kidnappers never really had any intention of releasing their US and British captives. Instead they were aiming for maximum psychological impact by killing them on camera. In those situations, armed intervention, if they can be located, is usually the only option available.

Hamas is something of a hybrid case. The atrocities committed in their raid into southern Israel on 7 October were gratuitously cruel and violent. Yet they have subsequently been willing to bargain the release of more than 100 hostages.

Today, Israel's hostage relatives are all too aware that nearly all those already released won their freedom not by daring military action but by painstaking negotiations and mediation by Qatar and Egypt, backed by the United States. To date, only one Israeli hostage, Private Ori Megedish, has been rescued by force.

But doing a deal with kidnappers, especially those considered by many governments to be terrorists, can be a bitter pill to swallow. Hostage-takers will demand something in return.

In the case of Hamas, that has meant releasing large numbers of prisoners from Israeli jails, pausing the fighting and significantly increasing the amount of aid getting into Gaza.

Yet any further successful return of the hostages, says Gen Herbert, can only come about through diplomatic means: "There is no effective military solution to this issue in Gaza."