I need to scrap the Mi Yodea site, to parse it as one Question and one Answer. I am part of ravgpt.ai, which refers in it's answer to the source.

  1. Should I use the title as the question, or take the question itself, or have a rule to take the question if it's less than X characters?
  2. Should I just take the answer with the highest score / only accepted / check if a comment has a higher rank. what do you think is best?
  3. Should I Use tags and for halcha tags bring only one answer, and other tags I should concatenate all answers above X votes?
  4. What is the value of X, when an answer has X votes I can assume the answer or comment is correct?
  5. PTIJ questions I need to skip. Is there anything else I should skip?
  6. Any question I need to ask myself before scraping?
  7. Any tips from brainstorming my task will be appreciated!
  8. (for programmers) Should I use the api or loop through url?

All thoughts are welcome! Thank you!

  • I recommend the tour. Discussions like this one will also be of interest.
    – msh210 Mod
    Commented Aug 22, 2023 at 16:40
  • 3
    Not always is the highest voted or accepted answer the correct one.
    – Menachem
    Commented Aug 25, 2023 at 14:58

1 Answer 1


The whole seeming thrust of this is to try to create a way to measure how good an answer is, and it seems based on the assumption that the best answer, or some other factors, will determine that an answer is correct.

You didn't ask what you mean by correct. If correct means the person who asked found it answered the question to their satisfaction, then that's at least in line with what SA is about, if not covering every case (or even the majority necessarily).

However, given that it seems that you are trying to train an AI aimed at "helping us understand traditional sources", we are left to guess that when you say "correct", you mean the answer is "the correct answer, as faithfully and authentically provided by the tradition". This is a big no no for SA.

SA doesn't have a haskama, and it is questions and answers from anons. There is no Rav on board showing every answer to their Rosh Kollel to verify it. You'll also be aware that the tradition is rich and varied, and there are often many answers that fit your criteria of "correct" on the same question.

So, I am not an AI programmer, I don't know if you can measure "trustworthiness" in how you rate the data you feed your bot, but I wouldn't know how to measure that for SA, but do note that it will certainly not be very high. Given what you are asking, I think it is impossible, and certainly not a good idea. If this is just an AI to look things up in a chat-like way, then fine (and if so, what makes it different to bing?), but if it is actually trying to understand this data as some sort of "Rav.AI" then my answer applies.

If you clarify the question in such a way that I feel you are asking something that can be answered, and takes into account the facts about how this site works, and how they affect your goals, I will happily retract my downvote and perhaps give an upvote.

  • Thank you for the answer, and clarification request. I meant for getting a correct answer, that a Rosh Kollel would say it's correct. I believe that crowd wisdom can achieve it. I don't know what SA stands for.
    – AJ AJ
    Commented Nov 11, 2023 at 17:47
  • 1
    @AJAJ stack exchange. If that's your goal, you are in the wrong place - note the big fat sign on every page saying that the standards here are not good enough to be deemed worthy of replacing consulting a Rabbi, not to mention a Rosh Kollel
    – Rabbi Kaii
    Commented Nov 23, 2023 at 19:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .