How effective is the weekly topic challenge proposal in generating new questions?

Question

How effective is the weekly topic challenge proposal in generating new questions?

Is there any way of measuring or estimating this?

Answer 1

The topic challenge ran continuously from 5772-03 (November 2, 2011) to 5772-23 (March 21, 2012). It was then briefly resumed from 5772-40 (June 18, 2012) to 5772-43 (July 24, 2012). For the purposes of this analysis, I'm going to focus on the 20 weeks from November to March and compare them to the previous 20 weeks (baseline) and the following 20 weeks (next). I'm using this query, which is the same type of data I used to analyze a contest for Parenting.SE.

q_type   askers questions answers q_edits a_edits q_votes a_votes q_comments a_comments touches_per_q 
-------- ------ --------- ------- ------- ------- ------- ------- ---------- ---------- ------------- 
baseline 126    785       1811    1732    1163    5974    8406    2129       3796       31.86         
contest  141    1102      2287    2481    1520    8391    10850   4264       6441       32.88         
next     189    937       1593    2197    1199    6088    6966    4835       4625       29.35  

As you can see, the first contest period had 317 more question than the previous period and 165 more than the next period. (Note: the next period overlapped with the other contest period.) The increase in askers from 126 to 141 to 189, suggests that more people were using the site in each successive time period. Assuming a rising tide, we can still estimate that the contest added a little over 8 questions a week. This seemed too many, so I tabulated the results of each contest as reported in the meta answers:

7: Weekly topic challenge 5772-03 (week of Lech-l'cha 5772): aliya laaretz

23: Weekly topic challenge 5772-04 (week of Vayera 5772): Shabbat songs

18: Weekly topic challenge 5772-05 (week of Chaye Sara 5772): Navi: books of the prophets

4: Weekly topic challenge 5772-06 (week of Tol'dos 5772): Ribbis: loans at interest

5: Weekly topic challenge 5772-07 (week of Vayetze 5772): Shidduchim: finding a spouse

4: Weekly topic challenge 5772-08 (week of Vayishlach 5772): Asked by children

18: Weekly topic challenge 5772-09 (week of Vayeshev 5772): Chanukah

4: Weekly topic challenge 5772-10 (week of Miketz 5772): Commentaries

4: Weekly topic challenge 5772-11 (week of Vayigash 5772): Violence

2: Weekly topic challenge 5772-12 (week of Vaychi 5772): Ikkarei Emunah: Fundamentals of Belief

3: Weekly topic challenge 5772-13 (week of Sh'mos 5772): Korbanot

5: Weekly topic challenge 5772-14 (week of Vaera 5772): Midrashim and Aggadot

3: Weekly topic challenge 5772-15 (week of Bo 5772): Kishuf (Magic)

3: Weekly topic challenge 5772-16 (week of B'shalach 5772): Light

3: Weekly topic challenge 5772-17 (week of Yisro 5772): Trees

4: Weekly topic challenge 5772-18 (week of Mishpatim 5772): Unusual Blessings

1: Weekly topic challenge 5772-19 (week of T'ruma 5772): Practical Kabbalah

1: Weekly topic challenge 5772-20 (week of T'tzave 5772): Yevamot

1: Weekly topic challenge 5772-21 (week of Ki Sisa 5772): Medieval Jewish History

2: Weekly topic challenge 5772-22-23 (week of Vayakhel-P'kude 5772): Pets

2: Weekly topic challenge 5772-24 (week of Vayikra 5772): colors

6: Weekly topic challenge 5772-25 (week of Tzav 5772): Book division

Adding those up, I get 125 contest questions (or just over 6 a week). So we saw more questions than just those asked in response to the contest. This suggests that the contest is purely additive and perhaps even slightly multiplicative. It's also clear that interest in the contests waned toward the end of the period.

Here's a quick look at the second contest period:

q_type   askers questions answers q_edits a_edits q_votes a_votes q_comments a_comments touches_per_q 
-------- ------ --------- ------- ------- ------- ------- ------- ---------- ---------- ------------- 
baseline 71     246       433     598     332     1712    2078    1412       1328       32.09         
contest  80     272       428     647     340     1622    1636    1412       1244       26.94         
next     75     255       429     571     310     1372    1588    1347       1226       26.84        

The results of this shorter period are incomplete:

2: Weekly topic challenge 5772-40: Pirkei Avot

?: Weekly topic challenge 5772-41 (week of Pin'chas 5772): Rashi

?: Weekly topic challenge 5772-42-43 (week of Matos-Mas'e 5772): grains

The statistics suggest that 17 extra questions were asked as a result of the contest (though I don't know if that's statistically significant).

Finally, here are the statistics for the entire period from November 2, 2011 to July 24, 2012 compared to the equivalent proceeding and following weeks:

q_type   askers questions answers q_edits a_edits q_votes a_votes q_comments a_comments touches_per_q 
-------- ------ --------- ------- ------- ------- ------- ------- ---------- ---------- ------------- 
baseline 186    1449      3515    2871    2007    9907    15959   3463       6826       30.74         
contest  249    1923      3694    4407    2574    13856   17192   8468       10535      31.58         
next     335    1972      3055    4060    2014    8850    10969   8669       8008       23.14         

I wouldn't say this is definitive evidence that the contests drove questions since the next period improved upon the contest period. But the statistics proved, when combined with the data above, supporting evidence that contests drove engagement of various types. It would also seem that artificially inflating question rate caused no particular harm in terms of letdown or distraction.