We might consider exclusion for building ZIP archives on GitHub (see https://www.reddit.com/r/PHP/comments/2jzp6k/i_dont_need_your_tests_in_my_production and symfony/symfony: Issue #6605). But obviously this has pros and cons we should look into.
I started removing my test files (also from libraries) from deployments long ago for this reason. I think removing the build stuff (like Build/ and Scripts/ in Neos), but also the non-compiled sources would make sense to me. This would of course break the situation where someone disables loadMinifiedJavaScript in production, but that’s a bad practice imho anyway (and horribly slow in Internet Explorer 
). So if we document that feature extra clear…
I agree with the discussion on the symfony issue (https://github.com/symfony/symfony/issues/6605)
It’s more a deployment problem, so we can provide a nice documentation how to exclude those file with rsync exclude per example and keep the archive with tests (test are important, people can read them, run them … maybe)