$25! = 1 \times 2 \times 3 \times \cdots \times 25$ includes all primes up to 25 as factors, and their products.

Check composite numbers in order:

• $26 = 2 \times 13$: both 2 and 13 are $\leq 25$, so $26 | 25!$. ✗
• $27 = 3^3$: $25!$ contains $3^{10}$ (plenty), so $27 | 25!$. ✗
• $28 = 4 \times 7 = 2^2 \times 7$: $25!$ has many factors of 2 and 7, so $28 | 25!$. ✗
• $\ldots$ continuing similarly through composites up to 57 — all divide $25!$.
• $58 = 2 \times 29$: 29 is prime and $29 > 25$, so 29 is not a factor of $25!$. Therefore $58$ does not divide $25!$.

$58$ is the smallest composite non-factor. Answer: $\mathbf{58}$.