1. Why does anything differ by many orders of magnitude?
Some cities are very big. Some are very small. This fact has probably never bothered you before. But when you look at how cities sizes stack up, it looks somewhat peculiar:
The X-axis is the size of the city, in (natural) logarithmic scale. The Y-axis corresponds to the density (fraction) of cities with that population. The peak is around the mark of 8 on the X-axis, which corresponds to $e^8\approx 3,000$ people.
You can see that the empirical sizes of cities almost perfectly matches a normal ("bell curve") distribution. What's the explanation for this? Is mayoral talent distributed exponentially? When deciding to move to a new city do people first take the log of the new city's size and then roll some normally-distributed dice?
It turns out that this is solely due to dumb luck and mathematical inevitability.
Suppose every city grows by a random amount each year. One year, it will grow 10%, the next 5%, the year after it will shrink by 2%. After these three years, the total change in population is
$$1.10\cdot 1.05\cdot 0.98$$
As in the above graph, we take the log
$$\log\left(1.10\cdot 1.05\cdot 0.98\right)$$
A property of logarithms you may remember is that $\log(a\cdot b)=\log a + \log b$. Rewriting (2) with this property gives
$$\log 1.10+ \log 1.05+\log 0.98$$
The central limit theorem tells us that when you add a bunch of random things together, you'll end up with a normal distribution. We're clearly adding a bunch of random things together here, so we end up with the bell curve we see above.
2. Why charities might differ by many orders of magnitude
Some of Brian's points are about how even if a charity is good in one dimension, it's not necessarily good in others (performance is "independent"). The point of the above is to demonstrate that we don't need dependence to have widely varying impacts. We just need a structure where people's talents are randomly distributed, but critically their talents have a multiplicative effect.There are some talents which obviously cause a multiplier. A charity's ability to handle logistics ("reduce overhead") will multiply the effectiveness of everything else they do. Their ability to increase the "denominator" of their intervention (number of bednets distributed, number of leaflets handed out, etc.) is another. PR skills, fundraising etc. all plausibly have a multiplicative impact.
More controversially, some proxies for flow-through effects might have a multiplicative impact. Scientific output is probably more valuable in times of peace than in times of war. GDP increases are probably better when there's a fair and just government, instead of the new wealth going to a few plutocrats.
Here's a simulation of charities' effectiveness with 10 dimensions, each uniformly drawn from the range [0,10].
3. Picking winners
Say that impact is the product of measurable, direct impacts and unmeasurable flow-through effects. Algebraically: $I=DF$. By linearity of expectations$$E[I]=E[DF]=E[D]E[F]$$
So if two charities differ by a factor of say 1,000 in their direct impact then their total impact would (on expectation) differ by 1,000 as well.
This isn't a perfect model. But I do think that it's not always correct to model impacts as a sum of iid variables, and there is a plausible case to be made that not only do charities differ "astronomically" but we can expect those differences even with our limited knowledge.
Acknowledgements
This post was obviously inspired by Brian, and I talked about it with Gina extensively. The log-normal proof is known as Gibrat's Law and is not due to me.
Thanks, Ben. :)
ReplyDeleteWhen thinking about capabilities of a charity -- intelligence, experience, willingness to work hard, connections, etc. -- effectiveness can be multiplied. Just talking about people, I think it's fair to say that some are hundreds of times more impactful than others. But these impact differences should already be compensated at least to some extent by existing donors. If you're a world-class genius, why are you having trouble getting the funding you need, and why has no one else stepped forward to help you get that funding? Of course, in the extreme this becomes like a claim that there are no $20 bills on the sidewalk, but a more moderate form says the differences shouldn't be expected to exceed tens to hundreds of times.
That was capabilities. What about the impacts the charity has on the world? I don't think those can be said to multiply appreciably. Take as an example the interactions between peace and philosophical progress. (I'm choosing philosophy rather than science as you did because I'm concerned that science may be net bad to speed up, at least in certain domains.) Say the current level of peace P is Pc and the current level of philosophical progress Q is Qc. Even if you think the total goodness G is multiplicative -- G = P * Q -- then if you increase peace by some small amount p and philosophical progress by some small amount q, the total increase in goodness is just (Pc + p) * (Qc + q) = Pc * Qc + p * Qc + q * Pc + p * q. Because p and q are small, the interaction p * q is negligible, and the change is basically just a sum of the independent effects: p * Qc and q * Pc. The reason it's different at the level of personal/organizational multipliers is that there, one charity can actually have double the intelligence or work ethic of another. No charity can double the amount of peace in the world.
As far as the I = D * F point, this would be true if there was just one flow-through effect, but I think every charity has several significant flow-through effects, including just how they affect the flow of employees and donors from and to neighboring causes.
Thank Brian. Good points as always.
Delete1. Efficiency: your result follows by the definition of efficiency, so I agree with that. But I'm not as convinced that charity markets are efficient, even to within two orders of magnitude. For example. I spend a lot of time thinking about this problem, and I'm unsure what the most effective way to spend my money is. And if I can't do it, then how can anyone else :P
2. Great point. Let's suppose that flow-through effects are additive. (I don't think this matters, see next point.)
3. Multiple flow-through effects: my guess is that flow through effects are all multiplied by direct impacts. Handing out twice as many bednets should double the additional years of schooling you provide, etc. So even if you have $I=D\sum_i F_i$ I think it holds.
1. Most small investors know little about the stocks they invest in. Market efficiency doesn't require everyone to be smart, only that there be a few people who know about the unexploited returns.
DeleteThe analogy with the charity case: Not everybody has to know the best charity. All that's required is that some people in the neighborhood of the charity can say, "Hey, this is really effective; we should donate more and publicize this." And then locally, the better charity gets more donations than the worse ones. In some cases, really good charities will be publicized more widely, like on a TED talk or in the NYT or in a "top charities" review. Cream rises to the top. The main way in which this doesn't apply is if your values are atypical (e.g., it's hard to imagine VO being so widely publicized). But even then, informal mechanisms help transmit the information. I decided that THL and VO seemed like two of the best animal charities after I asked some friends in the animal movement for their opinions. If the charity is good, it should be able to find some good people to help promote it. This cream-rising system is not perfect, but we should at least expect that astronomical gains should have been exploited. Gains of tens to hundreds of times may be left on the table.
3. I'm using "flow-through effects" in a broader sense to include cross-pollination considerations too. I think with cross-pollination, you basically can't get more than a 100X differential for charities that neighbor each other, because the in-flows and out-flows of donors and labor are going to happen at nontrivial rates. Also, not all the flow-through effects have massive differentials. Training, research, donor cultivation, media publicity, etc. are not 1000X different between international-health charities.
Brian, Ben and I aren't sure we understand your argument that when you consider cross-pollination, the differences between charities can't be more than 100x. Do you mean that if there was a really good technique discovered by one organization, soon other orgs for the same general cause would start using it? And so in that way the orgs would sort of level out? (Ben's guess). To give the example of the farm sanctuary v. VO again, even if someone told the farm sanctuary that leafleting is a more effective technique, I doubt they would switch the whole focus of their org. They're main focus will still be taking care of the few animals on their farm.
ReplyDeleteIf by cross-pollination you just mean members/donors/staff moving around from one org to another in the same cause, it seems that there could easily still be orgs 100x better than others based on their general focus/ activities.
Hi Gina :)
DeleteBy cross-pollination I mainly meant the second meaning you listed: members/donors/staff moving around. Spreading around a really good technique would better fall into the "market efficiency" category, though probably my terminology is confusing. :P
As an example, suppose that 1/25 of people who are solicited by your local farm sanctuary later find their way to the main Farm Sanctuary organization. And suppose Farm Sanctuary is not more than 10 times worse than VO in cost-effectiveness. (This seems plausible because some of their campaigns are useful, and back in 2012 they were actually doing a significant amount of veg ads.) Then from this pathway alone you'd get not more than a 1/250 difference.
Of course, there are other network effects too: Employees and staff moving to Farm Sanctuary, as well as members/donors/staff at Farm Sanctuary moving backwards to local farm sanctuaries. As a result, the overall sign isn't clear, but it's unlikely these different considerations all cancel out. As you add more and more terms, the variance of the sum increases.
Cross-pollination across causes is lower, but everything comes with an opportunity cost. Some small fraction of people who do any really ineffective project might have gone on to do a much better project, or maybe a much more negative project. Again, it's unlikely these effects all cancel. It is plausible that the local orchestra might be hundreds of times less effective than a top EA charity, but probably not thousands of times. And the orchestra's sign might very well be negative.
I think I just have a different intuition about what I would call astronomically different charities. E.g. if one was "-5 badness" and another was "+400 goodness", the absolute value of the ratio would be <100, but intuitively I would say they are vastly different levels of effectiveness.
DeleteIf the local ineffective farm sanctuary never existed, maybe most of those people would have found a more effective charity to donate to/volunteer at. So it doesn't feel right to say they're ~1/250 as good as VO.
Yeah, there is still a big difference. :) I was mainly trying to tone down the extreme claims that some EAs make about charities differing by "thousands of times" and such. Also, the same forces that make a local farm sanctuary within 250 times of VO also make other animal charities even closer to VO (though of course the sign isn't always clear).
ReplyDeleteThe local farm sanctuary isn't necessarily 1/250th as good as VO; they might have a negative effect, as you suggest, by drawing away talented people from research, outreach, and advocacy. Or they might be a gateway toward better things. You'd have to investigate more details about how people have moved due to the charity to determine the net sign.
Great read thank youu
ReplyDelete