Actually the big bird will occasionally take a whack at tormentors, but normally the mid air maneuvers required to catch the little birds would cost them too much energy. Little birds are far more agile than the big one, and choose their attacks carefully. Small birds tend to fly above a larger bird and interrupt the big bird’s flight with a series of pecks to the neck and wings.
The behavior of small birds attacking a larger predator is called "mobbing." The smaller birds are trying to drive the bigger bird out of their territory. Blue Jays and American Crows mob birds of prey all year long; while other birds like mockingbirds, red-winged blackbirds, and common grackles primarily mob these big birds during the breeding season.
For the big bird, fleeing is more reasonable than expending unnecessary energy thwarting a small pest that is simply more versatile in air. However, if you're lucky you might witness a more experienced large bird using flight techniques such as barrel rolls to flash their talons at smaller birds or to get crows and ravens off their backs.