Generally, it's the baby birds making all the noise. You can wait a few more weeks until they are old enough to fly out on their own, or you can attempt to remove them yourself.
The nest is likely on the top of the damper, so you may be able to reach up and pull it out by hand. If the flue is made of square or rectangle clay liners, there is also a chance the nest was built in one of the corners of the liner. Using a garden hoe or something similar may be an option to scoop them out.
When I used to re-line chimneys, we would often run into this problem. We were almost always able to remove the birds without starting a fire, but you may have little option.
Get on the roof with a good flashlight. You may be able to see them and work out a plan of action. Once you've removed the birds, a chimney cap is a good idea.