Preliminary final results put the Christian Democratic Union (CDU) just below the 5% threshold, its worst showing in any German election.
The far-right Alternative for Germany (AfD) won with 38.2%, ahead of the Social Democrats (SPD) on 35.5%. The Left (6.5%) and the Greens (5.7%) also cleared the bar.
SPD incumbent Manuela Schwesig is almost certain to remain state premier.
Merz spoke at CDU headquarters in Berlin shortly after 6 p.m., when early projections still had his party scraping into parliament. He called the result a "catastrophe".
He said he wanted "to push our country forward" and that "reforms must be carried out", remarks widely read as a sign he will not step down. He took the stage alone, although other senior CDU figures were present.
The defeat follows the CDU's disappointment in Saxony-Anhalt on Sept. 6, and losses in Berlin the same day. Merz is at record levels of unpopularity and has been criticised even inside his own party. Observers say the run of setbacks could cost him the chancellorship.
German press blames Merz
German commentators were blunt. Der Spiegel's Roland Nelles wrote that anyone in the CDU who wants to remove Merz should "do it quickly, or fall silent". He called Merz a "transitional chancellor" and urged him to hand the party leadership to North Rhine-Westphalia Premier Hendrik Wüst, 51.
Süddeutsche Zeitung's Nicolas Richter wrote that the CDU "is now only a shadow of itself". He blamed Merz's often empathy-free speeches, and also a party that keeps questioning his leadership.
Bild demanded a change at the top. Editor-in-chief Marion Horn wrote that "Merz is dragging his party down". Handelsblatt said he was "buying himself time".
Die Welt's Ulf Poschardt took a different line, arguing that the Left's surge in Berlin poses a bigger problem for the middle class than the AfD does. "The bourgeois centre is dead", he wrote.
Several commentators saw a lesson in Schwesig's result. Mainstream parties, they said, can beat the AfD if they field energetic, empathetic candidates.
(jh)
Source: PAP