Ulamog's ability will necessarily resolve first. When you activate Tortured Existence's ability, the first step is putting it on the stack, then you choose its target, then you pay its costs, discarding Ulamog. After the activation process is done, Ulamog's ability goes on top of the stack (as all new objets on the stack do; you can only choose an order when multiple triggered abilities you control truly go on the stack at the same time). So it resolves first and shuffles your graveyard in your library, before Existence's ability can resolve and return its target to your hand.
Private Mod Note
():
Rollback Post to RevisionRollBack
I'm a former judge (lapsed), who keeps up to date on rules and policy. Keep in mind that judges' answers aren't necessarily more valid than those of people who aren't judges; what matters is we can quote the rules to back up our answers. When in doubt, ask for such quotes.
In general, if a triggered ability (such as Ulamog's last ability) triggers as you pay the cost of a spell or ability, the just-triggered ability will go on the stack above that spell or ability (here, Tortured Existence's ability) (C.R. 405.2) and get to resolve before it (C.R. 116.4). Thus, the Ulamog ability will get to resolve before the Tortured Existence ability does. And if the targeted card leaves your graveyard, it will be an illegal target for the Tortured Existence ability (C.R. 608.2b). See also this thread and this thread.
However, there is a distinction here between the discard/sacrifice being part of the cost vs. part of the effect. For example, Victimize sacrifices a creature you control as part of the effect. So, if you choose to sacrifice Ulamog to Victimize, Victimize is already resolving by the time the sacrifice occurs, and Ulamog's trigger can't interrupt that process. You would reanimate the two targeted creatures, and then Ulamog's trigger would be put on the stack. If any triggers occurred because of the creatures you just reanimated, you would choose the order between them and Ulamog's trigger; for example, if one of the reanimated creatures was Cadaver Imp, you could have its trigger resolve before Ulamog, and return a creature -- even Ulamog -- to your hand from your graveyard before shuffling the rest of the cards away.
Two Score, Minus Two or: A Stargate Tail
(Image by totallynotabrony)