More than half of the nation’s governors say they will not accept Syrian refugees in their states, but they may have little choice.

States cannot pick and choose which refugees to take in, federal officials with the Health and Human Services Department’s Office of Refugee Resettlement warned in a letter sent out last week. If they deny services to any group of refugees, states may risk losing resettlement funding altogether.

“States that continue to use ORR funding must ensure that assistance and services are delivered without regard to race, religion, nationality, sex, or political opinion,” ORR Director Robert Carey wrote. “States may not deny ORR-funded benefits and services to refugees based on a refugee’s country of origin or religious affiliation.”

The letter, dated Nov. 25, was sent out last week to refugee resettlement programs in 49 states and the District. Wyoming does not have a resettlement program.