This phase II trial studies whether adding pembrolizumab to olaparib (standard of care) works better than olaparib alone in treating patients with pancreatic cancer with germline BRCA1 or BRCA2 mutations that has spread to other places in the body (metastatic). BRCA1 and BRCA2 are human genes that produce tumor suppressor proteins. These proteins help repair damaged deoxyribonucleic acid (DNA) and, therefore, play a role in ensuring the stability of each cells genetic material. When either of these genes is mutated, or altered, such that its protein product is not made or does not function correctly, DNA damage may not be repaired properly. As a result, cells are more likely to develop additional genetic alterations that can lead to some types of cancer, including pancreatic cancer. Immunotherapy with monoclonal antibodies, such as pembrolizumab, may help the body's immune system attack the cancer, and may interfere with the ability of tumor cells to grow and spread. Olaparib is an inhibitor of PARP, a protein that helps repair damaged DNA. Blocking PARP may help keep tumor cells from repairing their damaged DNA, causing them to die. PARP inhibitors are a type of targeted therapy. The addition of pembrolizumab to the usual treatment of olaparib may help to shrink tumors in patients with metastatic pancreatic cancer with BRCA1 or BRCA2 mutations.