President Barack Obama is moving ahead with his push to close the Guantanamo Bay prison despite the uproar over the exchange of five Taliban prisoners for a captured American soldier, an administration official said Thursday.

The government has been working to reduce a backlog of prisoners already approved after a security review for transfer to their homeland or repatriation elsewhere, the official told reporters.

The official said a “significant number” of prisoners are on their way toward release, but he declined to say precisely how many or when they would leave Guantanamo. The remarks were made on condition of anonymity amid fierce criticism in Congress over the decision to swap the five Taliban for Sgt. Bowe Bergdahl.