Apparently, there is an unwritten rule that the incumbent prime minister is given the first chance to form a government. That is typical of British constitutional principles. The so-called "British Constitution" is not a single written document, as is the case with most other countries, including the United States. Instead, it is a set of precedents, some of which are codified by acts of Parliament.
An article in today's Financial Times addresses how those precedents might apply to this situation:
Some Whitehall old hands are puzzled that both David Cameron and Nick Clegg have appeared to question the convention that allows Gordon Brown, as the sitting prime minister, the first opportunity to form a government. “It is absolutely the correct interpretation [of the rules] and if they feel that’s a rum deal they can challenge that in parliament,” said one who pointed out the practice was intended to ensure continuity.
That points up the tricky nature of dealing with an unwritten "constitution".
The article goes on to describe the roles of leading permanent civil servants in the process:
Sir Gus O’Donnell, cabinet secretary, is reshaping Whitehall practice to facilitate discussion while protecting the monarch, the integrity of the civil service and continuity of government. He is one of a “golden triangle” of senior officials – that includes Sir Jeremy Heywood, the Downing Street permanent secretary, and Christopher Geidt, private secretary to the Queen – with crucial behind-the-scenes roles in ensuring a smooth and stable transition after election day should there be a change of government.
One key consideration from their standpoint is:
ensuring the Queen is not “dragged into politics”. “She’s very loathe to be put on the spot,” said one person familiar with the conventions. “The strength of the system is that Her Majesty never has to use any of her latent powers.”
As I described here, a non-political head of state, such as Queen Elizabeth II, might intervene if there were a major crisis threatening her country's democracy. The need to negotiate a coalition government, while being an unfamiliar ritual to the British, would not constitute such a crisis. The Queen does not want to decide who the next prime minister will be, if it's at all possible for her to avoid doing so.
As a practical matter, coalition-building would largely involve the Liberal Democrats, and their leader Nick Clegg, deciding to back either the Conservatives or Labour. Some reports have indicated that Gordon Brown's resignation will be the price Labour will need to pay for a coalition. There are hints that Clegg may be willing to work with a successor to Brown, perhaps Foreign Secretary David Miliband, or Ed Balls, the secretary of state for children, schools and families. On the other hand, I don't think that a Liberal Democrat-Conservative coalition can be ruled out.