A number of countries are planning the use of “immunity passports” as a way to ease restrictive measures and allow infected and recovered people to return to work during the COVID-19 pandemic. This paper brings together key scientific uncertainties regarding the use of serological tests to assure immune status and a public health ethics perspective to inform key considerations in the ethical implementation of immunity passport policies. Ill-conceived policies have the potential to cause severe unintended harms that could result in greater inequity, the stigmatization of certain sectors of society, and heightened risks and unequal treatment of individuals due to erroneous test results. Immunity passports could, however, be used to achieve collective benefits and benefits for specific populations besides facilitating economic recovery. We conclude that sector-based policies that prioritize access to testing based on societal need are likely to be fairer and logistically more feasible, while minimizing stigma and reducing incentives for fraud. Clear guidelines need to be set out for which sectors of society should be prioritized for testing, and rigorous mechanisms should be in place to validate test results and identify cases of reinfection.