Scripting challenge

Question

I wonder how hard it would be to find all citations to a given set of texts referenced on this site and post them for reference somewhere on the web.

Would anyone think this is a useful endeavor? Would anyone be interested in doing this?

I would imagine this presented as an index to Mi Yodeya organized by sefer and daf/chapter/verse/siman/halakha/etc.

Some proposed use cases:

Let's say you're learning (or have just learned) Hilchot Tshuva. You look up Hilchot Tshuva in the index and see a bunch of citations on Mi Yodeya. Are there questions that have been asked about Hilchot Tshuva that you can now answer? Are there answers citing Hilchot Tshuva that you can understand more thoroughly or perhaps reject? Are there different perspectives on Hilchot Tshuva that you can appreciate more or less now that it's fresh in your mind?

Another use case is preparing for a vort on the parasha. Not every citation gets into tags (often it's in the answer).

There are probably other uses I haven't considered. Would this be useful for folks?

Answer 1

I suspect that parsing citations automatically from the text of posts would be too difficult, as they can look like "the next perek" (with no link to Sefaria or the like, which the script could parse) or "the same gemara cited in this answer".

So either we're to be left with either an incomplete index (which may be the best idea) or we need a solution to get around the oddly-formatted-citations issue. Here are some ideas for the latter:

• As we come across them, or maybe a script can find them to some extent, manually edit any post that has such a citation to have a real citation instead, or to have a special SGML comment that indicates it shouldn't be removed, is formatted specially for the indexing script, and includes a citation.
• Manually add stuff to the index, and have the script, on its next index-generating pass, not remove manually added index entries except in the case of deleted posts.

Answer 2

Some of the guys at the Hermeneutics site have started working on a Bible verse indexer and searcher for Stack Exchange sites, including Judaism. We're primarily focused on making a search script, but it wouldn't be hard to take the data and turn it into a reference->posts index.

https://github.com/alerque/stack-verse-mapper

Here's the current Judaism site index (which will continue to be refined as we develop the software): http://alerque.github.io/stack-verse-mapper/data/judaism-index.json

Answer 3

I began writing this and realized, after writing a directory structure with terms like lexer and indexer, that I'm basically rewriting a minor vertical search engine. Maybe there is something off-the-shelf that can do what I'm trying to do here.