Database management systems (DBMSes) depend on query optimizers to transform a user's declarative query into an efficient execution plan. Query optimizers are critical because a bad query plan can be orders of magnitude slower than the optimal plan. Modern query optimizers are complex and expensive to maintain, as they integrate a wide range of hand-tuned heuristics and manually-engineered cost models which must be updated for every new capability added to the DBMS. I will present two recent approaches to query optimization that leverage deep reinforcement learning to simultaneously improve query performance and decrease maintenance burden. The first approach, Bao (SIGMOD '21), learns to "steer" an existing query optimizer by training an agent via a contextual multi-armed bandit framework. The second approach, Kepler (SIGMOD '23), aims to find optimal query plans for parameterized queries using an exhaustive search process. More broadly, both Bao and Kepler highlight the huge potential impact of applying machine learning to systems problems, giving us a glimpse of what a fully learned system could do, as well as highlighting several potential pitfalls along the way.