That sounds suspiciously like something that we already tried, and it didn't turn out so well.
I presume you are talking about the Union of Soviet Socialist Republics. Actually, while the Soviet economy did stall, it did not utterly fail until after Gorby started his reforms in 1988. The Soviet recession of the 80s wasn't any more dire than the one in Japan that's been going on since the same time the USSR collapsed in 1991, or the global one that's been going on since 2007.
What really caused the USSR to collapse was the revival of nationalist and religious tendencies, moreso than a desire for Big Macs.
However, say we managed to implement such a scheme. Would we have a mechanism to fund risky projects? What about people who can't (or won't) work and innovate? When robots are better than humans at just about everything, will there be increasing pressure to kill unnecessary humans because they are placing too much strain on the planetary ecosystem?
Yes, it's called crowdfunding. When all you have to lose is just a few bucks, people will be much more willing to take risks. Venture capital is actually quite limited in the things it can innovate, since even speculative capitalists won't fund things that might benefit humanity greatly but aren't very profitable. And of course there are public projects as well.
People who don't work should be able to eat and survive since there is a surplus of resources, and it's the humanitarian thing to do. You also have to remember most people who don't work desire to, but are priced out of the labor market. It's not just because they are lazy like a lot of social Darwinist right wingers believe.
I think they will use Malthusian theory as an excuse to kill people off, yes, if things get to that point.
I agree that we need to avoid the rupture of society into a small cadre of elites and a huge mass of the poor. This is thoroughly underway at the moment, and has a lot of political support, so it's not just a hypothetical future scenario but rather something that we need to start reversing now. I think there are too many humans for the planet to safely carry, and we need to lose about 90% of them. Preferably this will be done by attrition, but some sort of pandemic, either natural or engineered, may get them first.
I don't subscribe to the Gaia hypothesis. It's just another religious superstition in my opinion. While it's useful to metaphorically talk about "Mother Earth", I don't believe this planet is actually sentient. There's no evidence to suggest there are regulatory mechanisms that prevent a species from getting to a certain number of individuals, if they are as smart as humanity. Rabbits might reach a peak population before they eat all their vegetables and die off, but humans are different since we can innovate and make barren land green.
Hoping for it to happen by attrition in a reasonable amount of time is unrealistic. Half of the 7.1 billion people alive now were born after 1983, so in 50 years there will still be about 3 billion people given life expectancies do not change even if
nobody ever gave birth to a kid again. That's still 6X more than the 500 million that's cited by the Georgia Guidestones as a sustainable population for the planet.
It will indeed take a pandemic unlike the world has ever seen: worse than the Black Death several times over, or a war that makes WW2 look like child's play, or a mass democide of the plebeians that makes the Holocaust look like peanuts, for the population to get back to the level deep ecologists and other assorted misanthropes wish for.