So, you have a dimple in the broken Allen wrench piece. Take a fine nail set, spring loaded center punch... Heck, even a finish nail and tap on it with a steel hammer hoping the vibration will unwedge the broken piece.
Alternatively, use the dimple to drill through the Allen wrench piece. Use a small drill bit and go slow with little to moderate pressure. Let the drill bit do the work. Push to hard, and it'll snap adding to the bad situation. That set of Allen wrenches you purchased are soft... They shouldn't twist off like that. Then, use the next size larger bit until you can get a screw extractor to bite well. Use little to moderate pressure removing the piece.
After the Allen wrench piece is removed, you may want to try tapping in the largest Torx bit you can to remove the set screw. Use a small amount of heat if no plastic is around. I couldn't tell what it is you are working on or the material surrounding it.
Take your time. Don't get frustrated.