Also, just to be clear, there can be social science fields which inquire into aspects of the engineering field of Computer Science, but that doesn’t effect the underlying character of the field.

Also, also, I would strenuously argue against the proposition that mathematics is in any way ‘human independent.’ But that argument is not entirely relevant.

If you read how architects justify their designs, they're full of references to culture, philosophy, and occasionally - too rarely - practical research into how people move through and live/work in spaces.

It's a social activity in that architects design for other architects. Clients are secondary.

CS is actually no different, it's just far less culturally self-aware. There's nothing objective about CS. It grew out of a specific philosophical current, and it could easily have gone in other directions, emphasising other kinds of entity relationships - e.g. less atomised and sequential, more contextual and associative. That in turn would have influenced the design of the hardware.

This is not true. The software field side also falls under computer science. Computer engineering is understood to mean the engineering of computers. Someone with a degree in computer engineering will have a strong electrical engineering background as well as digital circuit design, plus the lower-level side of software (operating systems, device drivers, general systems).

I’m assuming that by architecture you’re referring to system/enterprise architecture and not eg CPU architecture. For a given system, the viable solution space is massive, and the decision on which approach to use is a mixture of science (analyzing and meeting numerically-defined requirements), art (meeting soft requirements), politics (making everyone happy about the unquantifiable choices), and taste (choosing between different similar solutions when no other constraints makes one better than the other). Two organizations facing the exact same problem can come up with two vastly different solutions that both effectively solve the same problem.

How those decisions get made is, I believe, an extraordinarily interesting social sciences question!

There are huge seats of computer science that do, to me, fall under the umbrella of “hard science”, but not all of it. Philosophy, taste, and social dynamics play a huge role in “industrial computer engineering”, and to try to pretend that’s not the case is setting ourselves up for failure.

That said, I think the article mentions science being "social" in the context of how we weight scientific research depending on the authors standing and expression, not the absolute truth behind it. Which actually is probably quite a shame.

Studies can have interesting conclusions even in the presence of confounding variables. Sometimes the state of the art in a certain hypothesis space hasn't progressed past of of these yet, but that's still better than gut feeling.

In some areas like public policy research you will never completely eliminate all of these factors, but that's no reason not to attempt to create knowledge.

Smokers are routinely excluded from analysis of cancer rates, for a similar reason; the effect is so large that if (for example) you're looking at diet, and the difference in diet is correlated with a difference in smoking rates, any findings are invalid.

But fankly most of the posts here are people almost calling the original authors heretical for even trying to data mine commit messages.

I am half with you on this but thinking to SQL, its probably harder to find bugs in SQL than in an imperative language language, but its also a lot harder to create bugs - its a declarative language so doesn't allow you to create whole classes of bugs that you will see in an imperative language.

Personally I try to push logic towards the database for the very reason that it's less buggy in general in my experience.

> Science has its problems, but it’s still the best we got.

I would agree if you're doing particle physics and looking for p < 0.0000003 then absolutely. Science is a great tool for the kinds of investigations of the natural world which can be understood by science. If you're doing social science by scraping some data from the internet and looking for p < 0.05, that's a very different story.

I'm not actually sure what aspect of this qualifies as "science", besides using p-values to analyze the data. Even if all the categorization and sampling and analyses were perfect, it's using commit messages and known bugs with known fixes, all of which are essentially self-reported.

Philosophy had almost the opposite problem at one time ironically, the concern that theory was gaining precision but becoming unhelpful for formulating comprehensible statements (in the non-mathematical sense) about the human experience.

The greater problem is the politicalization of science for which I'll use Gender Studies, though this is not to say Gender Studies is the only politicized field nor that it is the most egregious offender in any way. Having a cohesive theory of gender is useful, especially for comparing and contrasting different cultures even if it's at a level that's superficial compared to other fields like physics. At the same time, some social movements have realized that they can use aspects of gender theory to grant themselves legitimacy while leveraging the exclusionary nature of academia to silence criticism and largely control the public dialogue.

A good example of the utility gained having a theory of gender is the various African refugee groups who have come to the United States over the years. I had the luck to interview a psychologist who did work helping to evaluate and integrate some of The Lost Boys of Sudan. Many of the Lost Boys exhibited signs of mental trauma and her role in helping them integrate into American life was trying to provide relief for this trauma. She noted that a major challenge initially was that there was common mindset, stemming from a mix of age and a group outlook rooted in the need to survive, where personal problems weren't viewed or even necessarily understood the way they are in the U.S. The result was that many of the refugees she was working with not only didn't see the point in many of her questions (and by extension how to answer them) they also had difficulty engaging with her at all in a manner condusive to therapy as it's practiced in the U.S.

Handwaving a lot for brevity, her initial approach was to explore a trait she had observed and viewed as a quiet but severe sort of masculinity that was extremely common amongst the boys she was working with. Making early progress was largely about coming to understand this quality better and speaking to it in ways that introduced new concepts by breaking them down into smaller ideas which could then be explored by discussing them as contrasts. Is 'quiet masculinity' an especially scientific descriptor? Probably not, but it was nonetheless useful to her as an ad-hoc compass concept for breaking down barriers to providing effective therapy and it was made possible by synthesizing her understanding of gender theory with her broader knowledge of human psychology.

Contrasting the previous anecdote is my experience with undergraduate Gender Studies courses which are often a mix of course work on theoretical fundamentals and lecture presentations. The presentations are rarely outright superfluous from what I've observed but it'd be dishonest to say that they aren't also laced heavily with normative political statements masquerading as sensibility and which offer nothing extra to the lecture in terms of information or utility. This sort of soapbox sophistry is unhelpful because when it's eventually brought to the spotlight it erodes the public's trust in academia which is often too compartmentalized and arcane for a layperson to see where one subfield ends and another begins.

This is, in my opinion, why the social sciences in general make for such an easy punching bag. Because it quickly becomes unclear why in one situation an idea is treated as rediculous while in another it might be presented as a key component of some broader framework.

The penultimate point to all of this being, it's because science deals with running into hard problems that it's important to recognize when we're mostly paving familiar territory and when we're operating mainly with ideas that are 'the best we've got'. The ladder is no less helpful in the face of total ignorance but it can be much more vulnerable to biases of perspective and culture. Vigilant introspection is therefore a constant necessity.

One day I encountered a phenomenon that is as real and physically dramatic as, say, the electric field of a Van de Graff generator. Being a fan of science I went looking for the math/science/engineering for this phenomenon. Long story short, after a few years of research, I discovered that: 1) this phenomenon has been known since antiquity in a pre-scientific way; 2) several researchers had independently (re-)discovered it multiple times going back about 2.5 centuries; 3) each time it had achieved a certain notoriety but then been debunked and subsequently ignored or forgotten. The first was Mesmer & "animal magnetism", but there are several others including Baron Von Reichenbach & the "Odic force", and Wilhelm Reich & "Orgone". This is repeating today with several kinds of "energy healing" including Reiki (which is distinguished by being Japanese and unabashedly theistic (Rei means Divine)) where practice is widespread, but lacking scientific validation. As I say, I've felt Reiki, so I know it's real, but so far I've personally never been able to interest any scientists in studying it, despite offering to demonstrate it. And the studies that have been done haven't helped. The Wikipedia article for Reiki just straight up calls it pseudoscience.

You know what we call alternative medicine that works? Medicine.

But anyway, well, see, there you go. Meaning no disrespect, my whole point is that you and others like you cannot or will not "see" this thing, yet it's definitely "there".

I think it's got to be the "social side of science" somehow preventing otherwise-rational minds from seeing what's right in front of them (even during a scientific study of Reiki.)

It's all Descartes fault, essentially, by postulating the "mind" as distinct from matter.

More specifically, placebo pain-relief has been associated with endogenously produced opioids for approximately 40 years now (1978, I believe).

More recent research suggests that much of the variance in the effects of pain-killers can be associated with these endogenously produced opioids (when the patient was not aware of the pain-relief, responses were similar).

So essentially, the placebo effect is just a normal part of your body responding to medical treatment, regardless of the efficacy of said treatment.

I'm talking about pain because most of the research is on pain, but the principles are similar where placebos tend to be effective (not against infections, sadly but pain and depression seem to be extremely placebo-responsive).

That's a good question. The subjective sensations correlated with various external events. What I mean is, I satisfied myself that I wasn't hallucinating.

> What sort of objective effect did it have?

Accelerated healing.

I've never tried to make a physical instrument to measure "chi", but the subjective effects of it on living tissue would be measurable with scientific instruments, I'm sure.

- - - -

edit to say: I don't want to hijack the thread but if you're genuinely interested AMA.

> Accelerated healing.

Clinical trials of products designed to accelerate healing are so difficult to do that few products are developed with such an endpoint. Your own experience may satisfy you, and that is great, but it's extremely difficult to set a baseline against which to measure any speed up in "healing".

I can hold my hand a centimeter away from another person's hand and "channel" Reiki to them and they can feel it clear up to their elbow. But I don't know how to make a scientific instrument to detect that.

(BTW, it is great! I was careless taking a pan out of the oven a few months ago and burned my thumb pretty good. I "beamed" it with Reiki and the pain went away and didn't come back. "So I got that going for me, which is nice."

Quoting myself from back then:

> I burned the tip of my thumb the other day taking a pan out of the oven, 450°F. I was careless and didn't quite hold the potholder correctly, burn reflex overridden so I wouldn't drop the pan, it melted the fingerprint whorls. After I finished hopping up and down, pissed off at my own carelessness, I took a few moments to "beam" Reiki to my thumb, while concentrating on communicating with my healing and immune responses telling them to "get in there and clean things up". The pain went away and didn't come back. A very modest blister formed under the melted skin. The spot was tender, and it stung a bit if I let hot water touch it the first day, but after that there was no tenderness. Two days later the blister was gone. The skin is slightly rough, but it's not peeling, nor is it a scab. Day three I woke up and forgot that I had burned myself. The flesh apparently reconstituted itself. I'm not sure it will even leave a scar. [It didn't.] The skin still hasn't peeled.

> In sum, a minor second-degree burn, apply ten seconds of Reiki, result: no pain and accelerated healing.)

I feel a certain ineffable something "filling me up" with a kind of immaterial pressure or presence. I could feel it within the volume of my body and (somehow) extending into the space around me. It's difficult to explain in words what it feels like, people describe both warmth and coolness, presence and emptiness, tingling and relaxation, sometimes in combinations. The best I can do is that it felt like a high-tension electric field if you could somehow feel the field directly. Anyway, she started at the base of my spine and went up the center line to the top of my head at which point I felt a flower- or fountain-shaped field extend out from the crown of my head into the space around my body. Simultaneous with this I experienced a deep psychological change, comparable in intensity to a modest dose of "magic mushrooms" or LSD, that was deeply pleasant and healing. At that time in my life I was suffering from severe depression and now it lifted and I felt wonderful for the rest of the evening. The effect wore off but the depression was never as severe again after that.

So that was my first experience with this thing called Reiki. Nobody had told me anything. I had no beliefs about it going in. And it sure wasn't the "placebo effect" (whatever that is.) I had no idea what to expect, or even to expect anything at all.

After that I found a group that practiced and taught Reiki without charging the traditional fees and worked with them. I also, as I said, went looking for anyone who had "done science" to it, with the results I mentioned above.

There's a lot of variation (in my experience) between different Reiki sessions even with the same people. Most are pretty mellow, and other than a deep and abiding sense of peace and health there's not much you can point to and say, "there, that's the effect".

(Really, all I want or expect is for folks to not insult me to my face for sharing the experience. It's not like I'm claiming I rode in a UFO or had dinner with Bigfoot, y'know?)

But the approach of counting commits on github just seems fundamentally flawed. The first step should be to show that the github dataset can be used to say anything about quality and productivity of things like language choice. Arguing about how to classify commit messages seem pointless without this foundation.

Is this even a good idea?

We’ve just spent several thousand words on methodology to show how the original FSE paper was flawed. But was that all really necessary? Most people point to a much bigger issue: the entire idea of measuring a language’s defect rate via GitHub commits just doesn’t make any sense. How the heck do you go from “these people have more DFCs” to “this language is more buggy”? I saw this complaint raised a lot, and if we just stop there I could have skipped rereading all the papers a dozen times over.

Jan Vitek emphasizes this in his talk: if they just said “the premise is nonsense”, nobody would have listened to them. It’s only by showing that the FSE authors messed up methodologically that TOPLAS could establish “sufficient cred” to attack the premise itself. The FSE authors put in a lot of effort into their paper. Responding to that all is “nuh-uh your premise is nonsense” is bad form, even if the premise is nonsense. Who is more trustworthy: the people who spent years getting their paper through peer review, or a random internet commenter who probably stopped at the abstract?

It all comes back to science as a social enterprise. We use markers of trust and integrity as heuristics on whether or not to devote time to understanding the claim itself. This is the same reason people are more likely to listen to a claim coming from a respected scientist than from a crackpot.12 Pointing out a potential threat to validity isn’t nearly as powerful as showing how that threat actually undermines the paper, and that’s what FSE had to do to be taken seriously.

Everyone believes that they occupy the intellectual high ground, and that it's everyone else who's guilty of confirmation bias and wooly thinking. Therefore, anybody else's accusations of confirmation bias and wooly thinking are merely a product of their own confirmation bias and wooly thinking, and can safely be ignored.

Attacking the methodology mitigates this effect. By demonstrating that they spent serious time digging into it and trying to replicate it, the authors give a strong signal that they're not just exercising in armchair skepticism. This, in turn, lends rhetorical weight to their attack on the premise. Without it, it would be as easy to ignore, and as quickly forgotten, as your average comment on Hacker News.

How do we know the premise is flawed?

> The first step should be to show that the github dataset can be used to say anything about quality and productivity of things like language choice.

How do you concretely do that? The initial paper, even if flawed, did a lot of work to try to control for confounding factors. To do what you said, you would have to put as much effort, if not more, to convince people that the results can be falsified to claim anything, with the same level of control in place.

It's not even clear that counting commits is entirely meaningless. Getting some signal out of it does seem tremendously difficult, but not impossible. So questioning the methodology is really the only way to rebut the experiment.

The post you're replying to includes an explanation addressing that point.

For this paper, the premise is that GitHub commit history indicates language quality.

The methodology is then the implementation of the study: how do we classify each commit? How do we identify the base language in a repository? What would lead us to discard a repository as a data source?

The methodology may be perfect, but you may still end up with nonsense because GitHub is not a comprehensive or representative source of data for studying programming.

The CACM paper then did a bad job of testing if this is true by not being vary careful with the labelling and cleaning of their data, a point on which they were then attacked.

Please notice that they were critiqued on their methodology, not for thinking that commits are casually influenced by the design of the language they modify.

I don't think you need to worry; this isn't "politically incorrect". It's just wrong.

Agree with what you said afterwards, though: Meticulous coders probably are less likely to write bugs. But that will be true regardless of the languages they choose or prefer.

It's something everyone has a strong opinion on, but nobody's ever really figured out how to measure. Leaving it a subject that's still firmly in the province of conjecture. Typically self-affirming conjecture.

Me, I like dynamic languages, but, given that this subject is pretty much a complete knowledge vacuum, populated only by occasional iffy, easily criticized studies like the one in question, I'm inclined to say that the only position that offers any decent chance of not being informed primarily by confirmation bias is to assume that, for the most part, none of us has any firm idea what they're talking about. Least of all oneself.

It already looks better if you consider that dynamic typing is more expressive and such languages take much less code to do the same thing and less time. This is not something that can be researched via commit messages, but it is rather obviously true, so the claim from the paper that "static typing is better" is also obviously false and is way out of paper's reach to even claim. But then again this is CS and such claims that go way beyond data and author's expertise are super common in CS papers. Personally I stopped taking CS papers seriously a long time ago.

I am inclined to disagree. Both static and dynamic languages suffer from the same problem here: The most well-known examples from each camp are warty messes that make terrible case studies, let alone poster children.

The dataset can be used to say how defect rates compare across programming languages, that's important given that in CS and PL research these things are usually claimed left and right without any evidence. What the dataset can't say is how that translates to quality and productivity, as that would require a lot more than just commit messages. But at the same time it's something people already have some rough ideas about, like that people writing in low level rigid inexpressive languages are less productive than in high level flexible expressive languages.

I'm a bit puzzled with the "not a dox" tweet, what is going on here? I know that this is probably the least interesting part of the article, but I'm confused and curious.

So if you look at two projects, and both have a bunch of bugs, that doesn't tell you much. You have to know bugs-per-unit-complexity, which is pretty damn close to unmeasurable.

It's telling, though, that the companies managing the world's most complex systems run them in C++, Java, and TypeScript. And more recently, Go and Rust.

With much finance research, if the results were real, the researchers would be making billions on Wall Street rather than be academics. So, too, with software research. If the results were real, the researchers could become gazillionaires advising FAANG instead of being academics.

Of course that's difficult to evaluate. But if their gains were in proportion, I think you'd see these crazy spikes like what you're saying about these researchers becoming gazillionaires. And it just doesn't seem to happen.

Which is not to say there's no difference—but 1) the improvements seem to be by smaller factors than what engineers believe 2) they don't seem to generalize as much as is often claimed: afaict, a significant aspect of the gains has to do with reducing impedance mismatch between problem domain and language (but advocates will typically adopt a stance that the language is just intrinsically superior).

I would be very interested to hear counter-examples if people have them (I know the PG/Viaweb/Lisp story).

I'm skeptical that the PG/Viaweb/Lisp story generalizes at all to non-geniuses. Like, sure, if you gave Paul Graham raw assembly he could probably do some pretty impressive stuff even so. Doesn't mean anyone else can. He's a genius. So is Robert Morris.

> They’re underselling the effect here. While the dominant factor is number of commits, language still matters a lot, too. If you choose C++ over TypeScript, you’re going to have twice as many DFCs! That doesn’t necessarily mean twice as many bugs, but it is suggestive. Further, while they say the effect is “overwhelmingly dominated by […] project size, team size, and commit size”, that doesn’t actually bear out. Only the number of commits is a bigger factor in language choice.

This is inaccurate. "Effect" is not the expected difference (i.e. difference between the means) but, roughly, the expected difference divided by variance. Just looking at the expected difference is insufficient to determine if the effect is large (and undersold) or small.

Expected difference is not an interesting statistical property, just as the mean isn't (by itself). If you're looking at ten Clojure projects and ten C++ projects, and all ten Clojure projects have 10 DFCs, while eight C++ projects have 8 DFCs and two have 500, then the expected difference is huge, but the effect is small. Indeed, when looking at the variance, Clojure, the "best"-performing language in this dataset, and C++, the "worst"-performing language in this dataset, the two were not very distinguishable, supporting everyone's finding of a very small effect.

Difference in means can meet the definition of effect size, and are listed as an example right away in the wikipedia article on it [0]! The key is that it capture the phenomena of interest (and generally, the magnitude shouldn't be a function of number of observations). In psychology, where scales often have arbitrarily defined ranges (eg IQ scales), means usually are not useful effect sizes.

The case you list is a separate issue from effect size (probably the violation of distributional assumptions in some implicit or explicit model). Even using difference in means / variance (say, cohen's d), two observations that extreme would make the interpretation of both mean and variance calculations pretty dubious (a problem not solved by dividing them).

https://en.wikipedia.org/wiki/Effect_size

> two observations that extreme would make the interpretation of both mean and variance calculations pretty dubious

Yeah, it's a bad example, but I think it at least gives an intuition for why expected difference is not a good measure of effect. This shows two data sets with the same expected difference but one shows a large effect and one shows a small one: https://imgur.com/a/hg2NNl2 (the rebuttal paper also draws the expected value with variance and shows a similar situation).

First paragraph: "Examples of effect sizes include the correlation between two variables,[2] the regression coefficient in a regression, the mean difference, or the ..."

> Who would consider the same expected difference to mean the same effect size when the population distributions are narrow and when they're wide?

For the case where the unit of measurement has a sensible, relevant interpretation (e.g. if a study measured dollar value of two interventions), and attempts to capture uncertainty via CI or another means, I would consider it one meausure of effect size.

The key to understanding effect size is that its most common focus is around making measures over arbitrary scales become scale invariant. But scales are not always arbitrary.

Daniel Lakens has a great article on ES and puts the motivation for calculating them well..

> First, they allow researchers to present the magnitude of the reported effects in a standardized metric which can be understood regardless of the scale that was used to measure the dependent variable. Such standardized effect sizes allow researchers to communicate the practical significance of their results (what are the practical consequences of the findings for daily life), instead of only reporting the statistical significance (how likely is the pattern of results observed in an experiment, given the assumption that there is no effect in the population).

https://www.frontiersin.org/articles/10.3389/fpsyg.2013.0086...

To the degree that the thing measured has an inherently meaningful scale to the researchers (eg sometimes dollars, time), then it is already an effect size measure.

There is some nuance here, since the level on which you might want to interpret something (eg the spread could be part of the value, especially to an individual who will have 1 and not many codebases).

You might also want to render it comparable to other studies that measure something else, and it's unclear how to convert it to your scale, so you use standardized ES measures (eg many meta analyses).

But an important point is that whether something qualifies is really a question of the scale. (What the best ES is for your specific question is another important issue!)

Yes, but later the article shows that "mean difference" is really mean difference divided by variance.

> an important point is that whether something qualifies is really a question of the scale

It's one of the concerns, but I would say it's the main one. An effect size needs to distinguish between the two cases here, both having the same expected difference: https://imgur.com/a/hg2NNl2

If you care about units, you can talk about the expected value/difference, but that doesn't make that a meaningful effect size. What you need to do in those cases is to, at least, mention both the expected difference and the variance.

I would say later the article shows examples of standardized effect size, so divides by variance. Whether that means effect size can't be a difference in means (and explains the wiki intro comment) I disagree with--see my Lakens comment for why.

> If you care about units, you can talk about the expected value/difference, but that doesn't make that a meaningful effect size. What you need to do in those cases is to, at least, mention both the expected difference and the variance.

This seems like it is begging the question. If you look at the definition and rationale for effect size, both in the wikipedia and in Lakens, a claim this strong is not there. For example, a CI over difference in means is one way to capture what a standardized effect size that uses a specific variance term might be doing (what Lakens describes as the second ES viewpoint: statistical significance, distinguishing between effect and power).

In any event, thanks for discussing--it's been really helpful to think about and revisit this topic!

If, however, you are interested in the effect on a single observation, then the variability in samples/populations needs to be taken into account.

If Github is used by ordinary programmers (what are the reasons to assume otherwise?) i.e., if the sample is unbiased then what is the issue with going from "these [random] people" to "this language"?

I think this science project would be more fun if assumptions were discarded. Just gather the data, do it without a hypothesis. Something interesting might come up worthy of one.

I liked the bit of crankology in the article. If you sound like a crank no one will take you seriously? I had one day pondered what if the cranks are right? How would we know? It follows that we can't know sounding crank equals being wrong and that our idea of "sounding serious" is based on a flawed data set, if it isn't noise entirely. The topic of the study here certainly doesn't inspire my confidence.

Am I the only one that thinks this is a naive assumption? They aren't really languages like English is a language--so much for intuition. Don't definitions matter to science? Does science carry a selfie stick? I'm suddenly keen on the antithesis of science or a less social science.

"FSE was a preprint: a paper that hadn’t necessarily gone through peer-review, editing, and publication in a journal. Academics share preprints because 1) the process of getting a paper journal-ready can take years, so this gets it out faster, and 2) academics can share their research with people who aren’t able to afford the ludicrous journal fees."

To my knowledge, all ACM conferences including the "ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering" are peer-reviewed. In CS, in my experience, peer-review for conferences is more thorough than journal review. Conferences are more important. The idea that they are preprints is something from linguistics or some other field.

"How did FSE take the rebuttal rebuttal? Not well.

"So that’s technically not a dox, because he didn’t publish Berger’s private information, but still. That’s a really asshole thing to do!"

I'm not sure I would call linking to someone's Facebook page from Twitter to be "doxxing" in any sense. Calling a fellow researcher a "donkey" is a bit of an asshole thing, too. (I have no link to said comment, but I know Emery Berger. He's very positive he's right, the smartest in the room, and very abrasive.)

Tl;dr: Software engineering research is a garbage fire. But we knew that.