We present COGNAC, a novel strategy for compiling quantum circuits inspired by numerical optimization algorithms from scientific computing. Observing that two-qubit entangling gates with small rotation angles tend to have lower error rates than the typical maximally entangling gates (like CNOT), we use a simple and versatile noise model to construct a differentiable cost function. Standard gradient-based optimization algorithms running on a GPU can then quickly converge to a local optimum that closely approximates the target unitary. By reducing rotation angles to zero, COGNAC removes gates from a circuit, producing smaller quantum circuits. We have implemented this technique as a general-purpose Qiskit compiler plugin and compared performance with state-of-the-art optimizers on a variety of standard benchmarks. Testing our compiled circuits on superconducting quantum hardware, we find that COGNAC's optimizations produce circuits with higher output fidelity and lower gate counts than existing optimizers while remaining competitive in compilation time.