Yes, it's to be expected. If you are maxing out your upload, then downloads will suffer.
Even when you are downloading, you send packets of info to the server you are downloading from, which tell it that you are still there, etc.
So take away 90% of that upload, and your downloads will be strained, as they fight to communicate with other servers.
Consider it more like a really fast conversation you are having, rather than a 'stream' of data. There has to be communication both ways.
There's not a lot you can do about this, routers sometimes hav "QoS" (quality of service) settings. If you can manually slow down the machine that is uploading, then the download should start to pick up again.
or wait...
I always have to explain to the others on my network that "It will be done uploading in 8 hours, chill out"