In theory you can put any binary files into Git, but people tend to avoid it since every clone of the repository will become quite large. I'd recommend only placing source code into the repository. Instead, host larger binary downloads like your bld-1.0.2.tar.xz files from some other location that's not a Git repository. Another possibility is to use GitHub's "Releases" feature, see here, I haven't tried it myself but it looks like you can attach binaries to releases. Note that since your current repo already includes the binary files in its history, the best way to get rid of them is to start a new repo with a clean history.

Also, I would recommend against putting other project's source trees into yours. You'll end up having to keep your repo up to date with the source repo all the time. Instead, look into externals (known as submodules in Git), see for example here. Another approach, useful for when the VCS systems aren't compatible, is to include in your project a script to download the other project's source code.