Undated (AP) _ A former American hostage said the death of the Ayatollah Ruhollah Khomeini means the ''nightmare is over'' for Iran, and former President Carter said he hoped the death would boost Iran from its hostility toward the West.

''I know he was a great hero in his own nation,'' Carter told reporters outside his church in Plains, Ga., on Sunday. ''My hope is that his successor will be more inclined toward peace and reconciliation.''

Radical Iranian students invaded the U.S. embassy in 1979 and held 52 hostages 444 days. Carter in many ways was the 53rd hostage since the crisis probably lost him re-election and the hostages were freed the day he left office.

President Reagan in turn was stung by Khomeini in an arms-for-hostages scheme to obtain the release of Americans held in Lebanon by pro-Iranians. The plan backfired and led to the major scandal of Reagan's administration.

Tehran radio reported early Sunday that Khomeini died in a hospital late Saturday in the Iranian capital at age 86.

President Ali Khamenei was named as his immediate successor, but experts said it was questionable whether Khamenei could wield the authority of the late leader of the Islamic revolution.

With different readings of what Khomeini's death will mean for Iran, the news provoked mixed feelings among the hostages.

''I'm not the type to say I'm happy he's dead,'' said former hostage Barry Rosen. ''But I do feel, to a certain degree, that that part of the nightmare is over for us and for Iran.''

''On a personal level, I said years ago I'd probably have a drink when Khomeini died,'' said retired Col. Charles W. Scott, another former hostage who lives in Jonesboro, Ga. ''But now I'm totally ambivalent.

''It doesn't change a thing. Khomeini was a symbol of the Islamic fundamentalist revolution and that's all.''

There are still enough people ''of his ilk'' in Iran to continue the fundamentalist theocratic state, Scott said.

''Obviously, I'm not one of his big boosters. But it doesn't serve any purpose'' to find satisfaction in Khomeini's passing, said Paul Lewis, of Champaign, Ill., a Marine guard when the embassy was seized.

Another former hostage, Kevin Hermening, 29, of Mosinee, Wis., was less restrained in his reaction.

''It is probably the best news that has happened to the world in a decade,'' he said. ''Iran can now move on to a post-Khomeini era.''

Rosalynn Carter, the wife of the former president, said she and her husband continue to feel some bitterness about Khomeini, but not enough to create joy at his death.

''I don't really have a firm reaction to it. Like, I could never say, 'Oh, I'm glad he's dead.' ... I don't have to judge him. God will do that.''