The ITulip Peer Review System

[Aetius Romulous]

There is general agreement amongst those who know - and those who don't - that our economy, governance, and social systems are under strain since the asset collapse 36 months or so ago. There is absolutely no agreement on why that is or what can be done about it and in addition, I think most would conclude there has been no practical attempt to tackle the problem.

We here at ITulip, as well as a zillion other internet sites discuss these things long and often. Some of us are idiots, misinformed, or soldiers of disparate ideologies. Some of us are intelligent, well informed, and sincerely looking to participate in solutions. In either case, I think we all realize that we will not be the people who effect policy decisions. That has traditionally been left to academia, or at least that has become tradition over the space of several generations.

Our future it appears rests on the shoulders of academics.

So the question I have is this; is academia the best place to develop the kinds of solutions we require? I have always felt no.

Just who are these academics on whom we rest our future? They are bright people for sure, however the academic "system" is completely gamed against the new, the radical, the outside the box - the brilliant. The bright people on whom we rely are not the best and the brightest of the group, but rather those that play the game of academics the best. The "players" if you will. Players who have every incentive to keep a very closed club that does not rock the boat. These are the folks who are peer reviewed, published, and cited. A self-perpetuating process that locks out the vast majority of study and thought, where thousands and thousands get buried in irrelevance to protect the few and fortunate "rock stars" who profit mightily from the closed shop of academia.

Where do the "thousands and thousands" go who can't get a gig at the NYT or a tenured teaching thingy at a major American university? Why they flee to the internet and become bloggers of course.

I mention this because of a now infamous exchange between two rival statisticians and bloggers that appears to have set a template for "on-line" peer review;

Nate Silver and Veronique de Rugy demonstrate how a more modern peer review process could work.

In addition, there has suddenly appeared a flurry of investigative reports and academic reflection on the weakness of the peer review system, a sample link below;

Expert Fiddling


So I'm curious to see how ITulip members feel about their ability to meaningfully participate in an alternative on-line peer review system, and what that system might look like. My position is that we are doing that right now, and any top to bottom reading of a serious economic thread here would stand up to any traditional peer review system - should it elect to accept a thread on Itulip. There is some serious intellectual weight here, as there is on hundreds of other sites around the world. Isn't it time we came up with a way to "stream" those countless and invisible threads, blogs, and comment sections into a forum for serious policy effecting debate?

The "screamer" sites and YouTube have scored the occasional success outing behavior that works its way into policy (Fat, stupid cops beating on skater kids with cell phones notwithstanding). The net has power. What about us?

[Spartacus]

Re: The ITulip Peer Review System

Reading some "rebuttals" occasionally reminds me of the old Saturday Night Live sketch where Jim Belushi plays a prosecutor at a witch trial.

His entire case consists of standing up, pointing at the Gary Kroeger character (the accused) and screaming at the top of his lungs "WIIIIITCH !!!!!!!"

Other times I just shake my head & wonder "why are you bringing THIS up AGAIN ????, And not presenting anything new, that was not discussed before???"

[FONT=Verdana][SIZE=2]There is general agreement amongst those who know - and those who don't - that our economy, governance, and social systems are under strain since the asset collapse 36 months or so ago. There is absolutely no agreement on why that is or what can be done about it and in addition, I think most would conclude there has been no practical attempt to tackle the problem.

If you want to introduce a new system & test it out on iTulip, note that there is a "system" of sorts available now, and it consists of
1. the "ignore user" feature
2. using your own judgment to not respond to posts, thus keeping cruft to a minimum

Where this has been done, like the moderated sections of USENET, the moderated areas eventually died out. It seemed that a lot of people, over the long term, for whatever reasons, preferred the wild-and-woolly unmoderated areas

It was generally agreed that in the short term the moderated areas had better discussions and stayed on topic. But in practice, they did eventually wither away.

It's the Yahoo vs Google dichotomy that Clay Shirky once lectured on, or the Google search versus the Google directory dichotomy. Sometimes search results filtered through an expert's eyes works better. Over the long haul, people preferred to get all of the raw, unfiltered data, including the crap, and filter it themselves, versus having experts filter it.

This isn't the exact speech that I remember, but it gives a flavour
http://dltj.org/article/clay-shirky-...ation-filters/

[ThePythonicCow]

Re: The ITulip Peer Review System

So I'm curious to see how ITulip members feel about their ability to meaningfully participate in an alternative on-line peer review system, and what that system might look like.

Interesting question, and good (i.e., I agree with them ) comments preceding it.

My experience with the Linux kernel colors my reaction and, as you will see, my suggestion.

The time may be ripe to produce an "open source" book on economics or other subjects of importance. I have in mind the following. One person steps up, presenting the early, incomplete, rough and modest draft of a book he'd like to work with along side others. The book draft is publicly visible in all its versions from that point forward. Those of us interested propose major and minor changes and additions. Some focus on typos and misspellings, some focus on some particular theoretical insight, some add new chapters, some help orchestrate the occasional substantial reorganization. Now and then the book is published to a wider audience in a new version, so long as sufficient interest is present. The original author maintains a final vote; if too few people find that person's contribution worth much effort, then the project whithers on the vine, or perhaps it "forks" and someone else who can gather wider support runs with it. The license is important; it must maintain copyright but require that changes are republished under the same terms (a GPL or similar license.)

[Rajiv]

Re: The ITulip Peer Review System

Spartacus,

I do not believe that you interpreted Aetius' proposal correctly. I think, his intention (correct me if I am wrong) was to lift Itulip's forum in order to critique policies and proposals that could have a chance at being implemented.

Currently those kinds of discussions occur among academics using a peer reviewed process. Also, policy is decided by academics, with relatively little input from outside academia and the corporate structure.

I believe his view is that at most times the quality of discussion at Itulip is better than, or rivals the discussions that go on at the policy making level. I believe his intent was aimed at trying to discover ways to raise Itulip to that level.

Any ideas?

[jpatter666]

Re: The ITulip Peer Review System

Interesting question, and good (i.e., I agree with them ) comments preceding it.

My experience with the Linux kernel colors my reaction and, as you will see, my suggestion.

The time may be ripe to produce an "open source" book on economics or other subjects of importance. I have in mind the following. One person steps up, presenting the early, incomplete, rough and modest draft of a book he'd like to work with along side others. The book draft is publicly visible in all its versions from that point forward. Those of us interested propose major and minor changes and additions. Some focus on typos and misspellings, some focus on some particular theoretical insight, some add new chapters, some help orchestrate the occasional substantial reorganization. Now and then the book is published to a wider audience in a new version, so long as sufficient interest is present. The original author maintains a final vote; if too few people find that person's contribution worth much effort, then the project whithers on the vine, or perhaps it "forks" and someone else who can gather wider support runs with it. The license is important; it must maintain copyright but require that changes are republished under the same terms (a GPL or similar license.)

TPC -- let's take Wikipedia as a starting point. What could be improved from there?

[oddlots]

Re: The ITulip Peer Review System

Aetius,

I am excited by the idea that we could collectively make some progress on some of the root issues that exercise us on these boards. This is mostly because I find myself churning away at economic conundrums that I've become familiar with over the last 5-6 years without seeming to make any real progress at - I won't say solving - let's say, clarifying them. Some of this is down to laziness, some of it to real time constraints, some of it to simple ignorance of fields I'm unfamilar with. That's my hobby horse at the moment, and I'll return to it in a moment. But first let me address some of your points.

Re. the idea that we can't trust the academics to sort it out: I agree. I even think it's more peculiar to economics than simply "academics." The abject failure of economics as a discipline in the current debacle strikes me as stunning really in its completeness and seems, if anything, under appreciated. In fact I think it only weakens one's perception of it to subsume it under the failure of a broader academic model. The failure belongs to economics as a discipline. Full stop.

The best, simplest expression of this I've come across was in a post by Yves Smith where she made the observation that in systems engineering efficiency is a secondary virtue while stability is a first order virtue: job one if you like. If that seems a bloodless description, think: "before you figure out how little steel you can use first figure out what's going to stop your bridge from collapsing."

You could tell the whole story of the financial bubble using that simple analogy and it'd probably do a better job of explaining it - while creating a minimum of pointless political "heat" - than most of the narratives out there. To get the joke you simply have to appreciate that market equilibrium theory simply assumes stability. And of course, since that's solved, we're free to start cutting away those pointlessly heavy supports. Capital efficiency (i.e., massive leverage) here we come!

My point in the above is not to criticise but to say I would really welcome a discussion of what each of us feels are the salient features of our predicament. Let's say the above - second-hand observation - is my opening contribution to the discussion.

[ThePythonicCow]

Re: The ITulip Peer Review System

TPC -- let's take Wikipedia as a starting point. What could be improved from there?

Interesting question. I have established and operated for several years a Wiki site (not a public site), and I am a long time user of Wikipedia. Wikipedia.com is one particular website using a form of web server called a Wiki server. Wiki servers allow end users to dynamically edit the presented web pages, and in particular provide a convenient way to name and add additional content pages linked to from the existing pages.

Wiki sites come in four flavors of success.

1. Wikipedia pages for non-controversial topics, such as the Python programming language or Cows provide an excellent quality, dynamically updated, wide ranging and easily searched equivalent to a traditional encyclopedia.
2. Wikipedia pages for controversial topics (I won't give examples here; cf. Rant & Rave) become highly politicized.
3. Lessor Wiki sites tend to grow stale and fade away, once whatever initial burst of energy focused on accumulating some particular volume of information dissipates.
4. Lessor Wiki sites can continue indefinitely if someone finds it worthwhile to pay to have them maintained, as one convenient mechanism for distributing some particular information, usually for a commercial or other larger purpose.

The problem with Wikipedia.com becomes visible with the second flavor above, the controversial topics. Successful open projects have one man (occasionally a female or a couple of men, seldom otherwise) who stakes his public reputation on the result and who has the smarts, temperament and drive to produce a good "product." He gets the last vote. This must be backed up with each of:

1. discussion open to all (at least anyone who will minimally behave themselves),
2. a permanently visible archive of both the discussions and of the changing revisions to whatever is the "product",
3. some clearly identifiable product the success of which can be used to focus efforts, and
4. a suitable open license that perpetuates these essential dynamics.

In the case of Wikipedia, the history of changes is sometimes expunged, the manner of decisions on controversial is rather opaque, and the key controlling decision makers half hidden. One ends up with an editorial decision making process that is not trusted on controversial topics, especially by those who disagree with Wikipedia on subject about which they feel strongly. This ends up in much the same place as peer reviewed literature. In both cases, stuff goes in, magic occurs, and stuff comes out which seems to reflect the biases of the magician in unaccountable ways.

I do not see (quite possibly due to lack of imagination) where yet another Wiki serves any purpose we might have at iTulip. I am unaware of any substantial new area of (intellectual, scientific, economic, philosophical, ...) area of investigation for which a Wiki proved valuable. Wiki's are more useful for organizing existing information, as a form of user editable content management system (CMS).

Some key things that Aetius Romulous alluded to in (my very loose reading of) his opening post that I find missing in any Wiki based site of which I am aware include

• the completely open review process (everyone sees all submissions immediately upon submission, before any non-trivial editorial control is even practical)
• the archiving of all comments and contributions (a by-product of the above)
• the systematic change history of the "product" itself, and
• (missing in Wikipedia.com, but not all other Wiki sites) the respected, actively present "benevolent dictator for life" (as Guido van Rossum likes to label himself; he's the creator of the Python programming language.)

[ThePythonicCow]

Re: The ITulip Peer Review System

The abject failure of economics as a discipline in the current debacle strikes me as stunning really in its completeness and seems, if anything, under appreciated.

Excellent rant.

However I do not find that significant advances in human understanding usually start with wide discussion.

Advancing human understanding is like mountain climbing in one regard; there will always be that one particularly dedicated and talented individual who gets there first. But it is unlike mountain climbing in another regard; once one guy finds a new peak, if any news of that discovery leaks out at all, it is common for many people of far more ordinary talent and less dedication to be able to climb that peak soon thereafter.

All significant software advances (and I imagine, no doubt parochially, all significant advances in our understanding of economics) begin with that one person who reached a new mountain peak first. Someone gets some new framework, some new insights, for understanding or working with something of interest. They flesh it out just enough that a few others who have already been working in the area can see the new concepts. Then the team work can begin.

[rjwjr]

Re: The ITulip Peer Review System

Still think we little people can influence the future of economic thought?...

http://prudentinvestor.blogspot.com/...-economic.html

Soros Pumps $50 Million in New Economic Think Tank

Tuesday, April 06, 2010

[ThePythonicCow]

Re: The ITulip Peer Review System

Still think we little people can influence the future of economic thought?...

yes .

[bpr]

Re: The ITulip Peer Review System

Excellent rant.

However I do not find that significant advances in human understanding usually start with wide discussion.

Advancing human understanding is like mountain climbing in one regard; there will always be that one particularly dedicated and talented individual who gets there first. But it is unlike mountain climbing in another regard; once one guy finds a new peak, if any news of that discovery leaks out at all, it is common for many people of far more ordinary talent and less dedication to be able to climb that peak soon thereafter.

All significant software advances (and I imagine, no doubt parochially, all significant advances in our understanding of economics) begin with that one person who reached a new mountain peak first. Someone gets some new framework, some new insights, for understanding or working with something of interest. They flesh it out just enough that a few others who have already been working in the area can see the new concepts. Then the team work can begin.

Methinks you're overthinking this, PCow.

A new software may not be necessary when an imperfect one already exists, and I think the solution was jpatter's question.

And you answered it masterfully.

A semi-closed-wiki is what's being proposed, where debatable points are discussed before being "published." The only question is whether the public version should include the debate that lead to the "final" version.

I'm not sure how to do this, but I'm sure it can be done to the existing PHP framework by limiting access to new edits to an editor, who can then publish them. The editor can even send edits to a board/forum before approving/editing them.

Aetius Romulous' initial post notes that certain individuals would be limited to certain tasks. I think a limited version (user-limited) of the Wiki platform could do this quite well.

[Aetius Romulous]

Re: The ITulip Peer Review System

I'm out battling the "real world" today, but I want to take the time to respond individually later tonight.

I do have to come clean and admit this was a bit of a fishing exercise. I am working on an essay that enumerates the issues with peer review and academic hierarchy, and proposes a working model of how the net can accomplish this readily and with the usual amount of entrepreneurial spirit. It was a "thought exercise", however my opinion is fast becoming that this is doable and the timing is right.
Opportunity!

I was looking for intelligent input and comment and so far I'm excited. So keep those cards and letters coming kids. Email me direct if you wish, but I really am excited about this. (easily entertained?)

Somebody is going to do this, right?

[Rajiv]

Re: The ITulip Peer Review System

Look also at "First Monday"

First Monday is one of the first openly accessible, peer–reviewed journals on the Internet, solely devoted to the Internet. Since its start in May 1996, First Monday has published 1,040 papers in 165 issues; these papers were written by 1,325 different authors. In addition, eight special issues have appeared. The most recent special issue was entitled A Web site with a view — The Third World on First Monday and it was edited by Eduardo Villanueva Mansilla. First Monday is indexed in Communication Abstracts, Computer & Communications Security Abstracts, DoIS, eGranary Digital Library, INSPEC, Information Science & Technology Abstracts, LISA, PAIS, and other services. First Monday's ISSN is 1396-0466.

Also look at the "post-autistic economics network" and their publication "real-world economics review" and their blog "Real-World Economics Review Blog"

[Serge_Tomiko]

Re: The ITulip Peer Review System

The net has power. What about us?

I've been using the internet for almost 20 years - since I before I was in high school or before Republicans took over Congress for the first time since my father was a child.

In 1992, it was the early days of politically correct indoctrination. Mass media culture reigned supreme.

The internet seemed to me to provide the gateway to overcoming the liberal indoctrination present in the media and to challenge the filtered educational information presented by teachers. The "information superhighway" they called it by 1994.

Yet, here we are today. The crowdism of mass culture has become even more pronounced, even on the internet. The tyranny of liberalism has gotten to the point where thought control legislation is present in nearly every Western Country, including your own. The academy exists for the sole purpose of maintaining the status quote of "equality". Even the grand debates in the political realm - such as Ross Perot - seem like distant memories.

I would argue that the power of the net has ultimately had no effect. Further, I would argue that the primary reason liberalism is advocated by the elite is due to the fact it eliminates social bonds, e.g.. religion, culture, race, etc. The internet has a profoundly antisocial effect on people which creates a similar state of being. In the end, you have a people who cannot organize effectively as they have no meaningful social bonds.

We should take the knowledge we have gained from sites such as this and educate those around us in the real world.

[Slimprofits]

Re: The ITulip Peer Review System

I've been using the internet for almost 20 years - since I before I was in high school or before Republicans took over Congress for the first time since my father was a child.

[..]

Yet, here we are today. The crowdism of mass culture has become even more pronounced, even on the internet.

I would argue that the power of the net has ultimately had no effect.

it's had a negative effect.