The centre-right CDU of Chancellor Friedrich Merz finished a distant second with 17.2 percent, followed by the SPD on 9.3 percent, the Greens on 8.9 percent, the Left on 8.6 percent, and the Sahra Wagenknecht Alliance (BSW) narrowly clearing the 5 percent threshold with 5.3 percent.
The AfD fell three short of a majority, winning 39 of the 83 seats in the state parliament, meaning it cannot govern alone.
The AfD and BSW together have enough seats for a majority, though no coalition between them has been declared.
BSW leader Sahra Wagenknecht has previously suggested her grouping could work with the AfD, though it remains unclear whether its local lawmakers would agree.
Alternatively, the CDU, Left, SPD and Greens could try to form a coalition to keep the AfD out of power.
AfD leaders say the final results are what matters most and remain hopeful of forming Germany's first far-right-led state government since the Second World War.
Poland's governing coalition reacted with alarm.
Prime Minister Donald Tusk wrote on social media that in Poland, "only idiots or traitors" could celebrate an AfD triumph in Germany, adding that "quite a few" such people could be found in the Confederation party and the Law and Justice (PiS) opposition.
Ryszard Petru, head of the Centre parliamentary group, called the result "a serious warning for the whole of Europe", arguing that when mainstream parties fail to solve problems around the economy, migration and security, voters turn to radicals.
He said Europe needed "less ideology and more effective answers to people's real problems".
Major setback for Merz's government
According to the Reuters news agency, the result is a major setback for Chancellor Merz, with his approval ratings sliding amid discontent over his economic reforms and communication style.
A recent poll by German public broadcaster ARD found that 87 percent of the state's voters were unhappy with the federal government.
Economist Marcel Fratzscher told Reuters the result reflected no confidence in Berlin rather than local issues.
Nationally, the AfD now leads polling at 28 percent to the conservatives' 20 percent.
Mainstream parties still refuse to work with the grouping, though it could find a partner in BSW.
The AfD's platform includes curbing immigration and pushing education and cultural institutions toward more traditional values, with proposals to teach the children of refugees in segregated classes and ban rainbow flags in schools.
It has also pledged citizen patrols, more Russian-language teaching and a halt to new wind farm construction.
(ał)
Source: IAR, Reuters, onet.pl