TIHANY, HungaryAt a tavern in this lakeside resort town, the waiter scurrying from table to table could use some help.
Most of his colleagues have gone abroad to seek better pay, leaving Ferenc Punk's family-run business heavily understaffed.
Mr. Punk could have tapped a vast pool of potential workers: Scores of migrants, mainly from Syria, Iraq and Afghanistan, have traveled through the Central European country over the past two years, and thousands more are waiting at its southern border in the hope of entering the European Union.
Instead, the 67-year-old restaurant owner plans to make his dining room smaller. He won't recruit migrants, especially if they are Muslims.
"I wouldn't know what to do with them," Mr. Punk said. "I'm not against them but they are coming from a totally different culture."
Mr. Punk's tavern is a microcosm of the wider forces shaping Hungary's society and economy ahead of Sunday's referendum on immigration. Prime Minister Viktor Orban, who opposes Muslim immigration, wants voters to say no to an EU policy to impose "the obligatory resettlement of non-Hungarian citizens."
Although Hungary's working population is expected to shrink by more than 10% over the next four years, according to a central-bank estimate, Mr. Orban objects to allowing large numbers of refugees to settle in the country, saying they would threaten its ethnic cohesion.
The prime minister argues migrants also represent a security threat because some Islamist radicals have made their way into the EU by posing as refugees.
In tandem with Poland's Jaroslaw Kaczynski, leader of the country's ruling Law and Justice party, Mr. Orban is campaigning in Brussels against the EU plan to impose refugee quotas on bloc members.
That has put him on a collision course with other EU​ leaders, notably German Chancellor Angela Merkel, who has made the acceptance of refugees a cornerstone of her policy.
But Ms. Merkel's approach has suffered setbacks at home, where an upstart anti-immigration movement outpolled her ruling conservative party in a recent state election and now has seats in 10 state parliaments.
"Courageous politicians in Germany and Austria do say that what the Hungarians are doing is good for Germany and for Austria as well," Mr. Orban said on state television Tuesday.
Since joining the EU in 2004, the country of nearly 10 million has seen an estimated 350,000 of its skilled and educated workers leave. At least 90,000 alone have relocated to the U.K. The "brain drain"along with a low birthrate and aging populationhas resulted in chronic labor shortages in service industries and at factories and farms.
Students, who were mandated to provide cheap labor at apple-picking camps during the communist era, are no longer coming to the fields. Romanians and ethnic Hungarians in neighboring countries traditionally filled many jobs in the agricultural and manufacturing sectors here, but now leapfrog Hungary to move farther west.
That has left members of DelKerTESZ, a cooperative of some 500 vegetable producers in Szentes, in southern Hungary, mustering friends and relatives to avoid losing harvests.
"We get family members to work on the fields or transport the produce on Saturdays and Sundays," said Ferenc Ledo, the cooperative chief. "It's impossible to find crop pickers."
Hungary's main association of entrepreneurs and employers, Mgyosz, has warned that the dearth of applicants in many sectors was endangering economic growth, which is expected to reach 2% this year. The association sides with government policy, though, saying thatonly migrant workers who "are skilled and could culturally fit in" are needed.
More than 150,000 non-European refugee seekers applied for asylum in Hungary last year. But nearly all of them continued their journeys to Germany or Scandinavia.
Those countries would like Budapest to repatriate any migrants who were first registered in Hungary upon their entry into the EUas the bloc's rules requirebut Mr. Orban has refused their entreaties.
The government now lets about 30 migrants a day cross into Hungary through its heavily guarded border with Serbia. Almost none of them stay in the country, even if granted asylum protection,according to the Helsinki Committee, a human-rights group.
The government's anti-immigration policy has been pilloried by the Two-Tailed Dog Party, a group founded by prankster artists that launched a mock billboard campaign.
"Did you know? Hungarians see more UFOs in their lifetime than migrants," one of the recent billboards reads.
Yet a survey conducted in mid-September by polling agency Publicus suggests 61% of Hungarians will support Mr. Orban's immigration stance on Sunday. The referendum, which is valid only if turnout surpasses a threshold of 50%, is largely symbolic, as it carries no legal bearing and Brussels has largely backed off trying to impose refugee quotas on bloc members.
At the Tihany tavern, Mr. Punk casts his support with the government, saying it has made Hungary a safer and more attractive place. "Some Germans have already moved here," he said. "They can have a cozy life here."
Write to Margit Feher at margit.feher@wsj.com