The way the system actually works varies from one monarchy to another.
In Spain, following a general election, the King, through the Speaker of the House of Representatives, actually has to propose a candidate to be the next prime minister . The proposed nominee is then put to a vote in the House and confirmed if he/she gets the support of a majority (more than half) of the members. If the proposed candidate fails to be confirmed in the first ballot, he/she can be confirmed in a second ballot by a simple plurality as long as the number of members voting against the nomination is not bigger than the number of those voting in favor of it (that's what just happened to Mariano Rajoy following the last Spanish general election BTW). If the two aforementioned ballots fail to confirm the PM , the King might propose another candidate , but, if 90 days from the first investiture vote elapse without a PM being confirmed, then the House of Representatives is dissolved and a new election is called.
Sweden uses a slightly different system where, after the election, if the sitting PM doesn't resign, a vote in parliament to confirm him/her in office is held (at most two weeks after the new parliament is convened). A confirmation only requires however that there is not a majority (more than half) of the MPs voting against it, which is sometimes called "negative parliamentarism" if I recall it correctly. If the PM resigns or is discharged and the office is vacant, the Speaker of the parliament proposes a new candidate to become PM and the nomination is put to a vote where, again, the proposed candidate succeeds if there is not a majority of MPs voting against him/her; note that, when abstentions are taken into account, the Swedish model could in theory deliver a PM that actually had fewer votes among MPs to confirm him/her than to reject him/her (as long as the reject vote doesn't get above 50 % + 1 of the MPs). If after four attempts, all proposed candidates are rejected under the rules above, then parliament is dissolved and a new election is called.
The UK, on the other hand, follows a different model, which has actually changed slightly since 2011. Unlike in Spain or Sweden, there is no compulsory confirmation vote in the House of Commons (HoC) for someone to become or stay as PM. A person holds the office of PM as long as he/she doesn't resign or is removed by the Queen (the latter situation happened the last time during the reign of William IV, so it's not really relevant today).
Basically, in the UK, if it is clear that the PM and his/her party lost the election, then he/she resigns and advises the Queen to send for the leader of the opposition to form a new government. Otherwise, the incumbent government, as the representative of the party with the most seats in the HoC (though not necessarily a majority) usually has a go on trying to form a new government. If the HoC later passes a motion of no confidence in the proposed new government, which requires only a simple majority (i.e. more MPs voting for it than against it), then the House has 14 days to pass a motion of confidence most likely in another alternative government, or otherwise, a snap election is triggered and the House is automatically dissolved (I believe, 25 working days before the date set for the snap election). In any case, the PM never leaves office until a new PM is appointed, which always happens in an audience with the Queen.
Those who understand the constitutional limitations, I'd love a little help getting a better grasp of how strong a role the monarch is allowed.
Now, I don't know enough about the DUP to have one opinion or another about the viability of them working with the Conservatives to make May's plan for a government work. I don't know enough about coalition governing, period, not being from a nation with a parliamentary system. So this question is totally hypothetical, not a statement on the current situation.
Say a PM's plan to work with another party to form the government raises a lot of eyebrows and a general questioning of whether the two groups can really work together as promised. If the PM goes into her meeting with the Queen expressing great confidence that she's sorted things out and can form a government, is the Queen constitutionally required to accept the PM's proposal? Or does she have the freedom to say, "no, I really don't think you've thought this through, you need to spend some more time with it and get it right?"
I ask because the situations I've seen described in the past where the monarch would say "no, try again" all hinged on the PM walking in and admitting that the deal may be shaky. In other words, a case where the monarch's refusal to accept the government was really an amplification of the PM's own concerns, and therefore not exactly the monarch expressing political opinion or "meddling."
But does the Queen have the power to draw a line in the sand if she observes problems with a planned coalition that the PM does not see or does not wish to acknowledge?