Area 51 analysis of committment phase

getRep[elem_]:=First@Cases[elem,XMLElement["span",{"class"->"reputation-score",__},{rep_}]:>rep,Infinity]

getName[elem_]:=First@Cases[elem,XMLElement["a",{"shape"->"rect","href"->ref_},{name_}]:>Hyperlink[name,"http://area51.stackexchange.com"<>ref],Infinity]

getPage[proposal_Integer,page_Integer]:=Module[
{ad,ct},
ad=Import["http://area51.stackexchange.com/proposals/"<>IntegerString[proposal]<>"?page="<>IntegerString[page]<>"&phase=commitment&committers=mostrecent",{"HTML","XMLObject"}];
ct=Cases[ad,XMLElement["div",{"class"->"committer-info"},___],Infinity];
ct=ct/.s_String/;StringTrim[s]===""->Sequence[]//.XMLElement[_,_,{}]->Sequence[];
{getName[#],getRep[#]}&/@ct
]

newProposalID=37304;
oldProposalID=15787;

Monitor[
page=0;
result=FixedPoint[Join[#,getPage[newProposalID,++page]]&,{}];,

StringForm["Loading page `` ...", page]
]

Committer count:

Length[result]

Committer count excluding duplicate names (could be an undercount):

Length@DeleteDuplicates[result[[All,1,1]]]

Grid[Sort[result],Alignment->Left]

Export["A51 committers.xlsx",result[[All, 1,1]] ]

scores=ToExpression/@(StringReplace[#,{"k"->" 1000",","->""}]&/@result[[All,2]])

Sort@Tally[scores]

Duplicates:

Cases[Tally[result[[All,1,1]]],{_,n_/;n>1}]

Load old proposal:

Monitor[
page=0;
oldresult=FixedPoint[Join[#,getPage[oldProposalID,++page]]&,{}];,

StringForm["Loading page `` ...", page]
]

People committed to the old but not the new proposal:

FullForm[oldresult[[1,1]]]

oldonly=Complement[oldresult[[All,1,1]],result[[All,1,1]]]

both=Intersection[oldresult[[All,1,1]],result[[All,1,1]]]

Length[oldresult]

Length[oldonly]

Length[both]

Grid[Sort[both],Alignment->Left]

Length[oldonly]

But of these, Heike Gramberg, Phil Maymin, David Burmaster, Rolf Mertig, Sseziwa Mukasa actually committed to both proposals, but under different names.

Grid[Sort[oldresult],Alignment->Left]

Beta site

getBetaPage[page_Integer]:=Module[
{ad},
ad=Import["http://api.stackexchange.com/2.0/users?page="<>IntegerString[page]<>"&order=desc&sort=reputation&site=mathematica","JSON"][[2,2]];
{"display_name","creation_date","reputation","is_employee","last_access_date","user_type","bronze","silver","gold"}/.#&/@ad
]

getBetaPage[page_Integer,site_String]:=Module[
{ad},
ad=Import["http://api.stackexchange.com/2.0/users?page="<>IntegerString[page]<>"&order=desc&sort=reputation&site="<>site,"JSON"][[2,2]];
{"display_name","creation_date","reputation","is_employee","last_access_date","user_type","bronze","silver","gold"}/.#&/@ad
]

Monitor[
page=0;
result=FixedPoint[Join[#,getPage[newProposalID,++page]]&,{}];,

StringForm["Loading page `` ...", page]
]

Length[result]

This is for the Mathematica site.

Monitor[
pageβ=0;
resultβ=FixedPoint[Join[#,getBetaPage[++pageβ]]&,{}];,

StringForm["Loading page `` ...", pageβ]
]

I use this one for other betas that I am comparing to.

Monitor[
pageσ=0;
resultσ=FixedPoint[Join[#,getBetaPage[++pageσ,"poker"]]&,{}];,

StringForm["Loading page `` ...", pageσ]
]

Check the results have come through properly

First@resultβ

Number of users that registered and never came back.

Length@Cases[DateDifference@@@Transpose[{DateList/@resultβ[[All,2]],DateList/@resultβ[[All,5]]}],0.]

Number of committers to the beta that have actually registered.

Length@Intersection[resultβ[[All,1]],result[[All,1,1]]]

Total number of users

Length@resultβ

For the comparator beta: number of users that registered and never came back.

Length@Cases[DateDifference@@@Transpose[{DateList/@resultσ[[All,2]],DateList/@resultσ[[All,5]]}],0.]

Histogram of user reputation on the comparator site.

Histogram[resultσ[[All,3]],25,Frame->True,PlotLabel->"User reputation",PlotRangePadding->{{0,0},{0,10}}]

Histogram of user reputation on the Mathematica site.

Histogram[resultβ[[All,3]],25,Frame->True,PlotLabel->"User reputation",PlotRangePadding->{{0,0},{0,10}}]

Cumulative user registrations.

DateListPlot[Transpose[{{2012-1942,0,0,0,0,0}+#&/@Sort[DateList/@resultβ[[All,2]]],Range[Length[resultβ]]}],Joined->True,PlotMarkers->{Automatic,Tiny},PlotLabel->"Cumulative users"]