How are you importing to a 560? Are you submitting to its web interface CWIS?
Or are you actually using a 560 with a Fiery RIP and importing direct to Command Workstation?
If you were printing the document to any printer with Acrobat/Reader it should reduce the size by 3% to account for page margins assuming you set it to A3 in Page Setup so that the preview window shows the correct 11.7*16.54
Check in the driver that it isn't trying to resize after the other settings are applied via Paper > Other Size > Advanced Paper Size
Your output heavily implies somewhere in the job something thinks there is 11*17 in use.
If you are importing into the Fiery and this happens, and it sees the A3 setting, your best bet would be to just increase the size by 5% at that point, if it is a job that will be run often setup a virtual queue with the enlargement already set , if you do it very often set it up as a hot folder and you can just drag and drop (Assuming you have the Hotfolder setup)
I could not recreate your issue on our EX560 Fiery or bustled Fiery 560, or the 560 itself via the driver submission.
If you are submitting through CWIS I would assume the printer is incorrectly identifying the size of the document for some reason.
My test file (created in Word 2010, saved as pdf for this test) is attached