Dyson spheres are hypothetical megastructures built by advanced extraterrestrial civilizations to harvest radiation energy from stars. Here, we combine optical data from Gaia DR2 with mid-infrared data from AllWISE to set the strongest upper limits to date on the prevalence of partial Dyson spheres within the Milky Way, based on their expected waste-heat signatures. Conservative upper limits are presented on the fraction of stars at G $\leq$ 21 that may potentially host non-reflective Dyson spheres that absorb 1 - 90$\%$ of the bolometric luminosity of their host stars and emit thermal waste-heat in the 100 - 1000 K range. Based on a sample of $\approx$ $2.7\mathrm{e}\,5$ stars within 100 pc, we find that a fraction less than $\approx$ $2\mathrm{e}\,-5$ could potentially host $\sim$300 K Dyson spheres at 90$\%$ completion. These limits become progressively weaker for less complete Dyson spheres due to increased confusion with naturally occurring sources of strong mid-infrared radiation, and also at larger distances, due to the detection limits of WISE. For the $\sim2.9\mathrm{e}\,8$ stars within 5 kpc in our Milky Way sample, the corresponding upper limit on the fraction of stars that could potentially be $\sim$300 K Dyson spheres at 90$\%$ completion is $\leq$ $8\mathrm{e}\,-4$.