This phase II trial studies how well the combination of avelumab with liposomal doxorubicin with or without binimetinib, or the combination of avelumab with sacituzumab govitecan works in treating patients with triple negative breast cancer that is stage IV or is not able to be removed by surgery (unresectable) and has come back (recurrent). Immunotherapy with checkpoint inhibitors like avelumab require activation of the patient's immune system. This trial includes a two week induction or lead-in of medications that can stimulate the immune system. It is our hope that this induction will improve the response to immunotherapy with avelumab. One treatment, sacituzumab govitecan, is a monoclonal antibody called sacituzumab linked to a chemotherapy drug called SN-38. Sacituzumab govitecan is a form of targeted therapy because it attaches to specific molecules (receptors) on the surface of tumor cells, known as TROP2 receptors, and delivers SN-38 to kill them. Another treatment, liposomal doxorubicin, is a form of the anticancer drug doxorubicin that is contained in very tiny, fat-like particles. It may have fewer side effects and work better than doxorubicin, and may enhance factors associated with immune response. The third medication is called binimetinib, which may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth, and may help activate the immune system. It is not yet known whether giving avelumab in combination with liposomal doxorubicin with or without binimetinib, or the combination of avelumab with sacituzumab govitecan will work better in treating patients with triple negative breast cancer.