Can the two PCIe interfaces both be used simultaneously to transfer the data to the GPU, so would this allow an effective transfer rate of 64 GB/s?
yes, this artificial example assumes that both can be used concurrently (which also is reasonable in practice).
Does the size 4096 of the square matrices refer to one side of the matrix (so 4096 x 4096 elements) or the total number of elements?
4096 is the number of rows and number of columns, so a matrix holds 4k x 4k elements.