Site talk:Developer stuff/Pages with content

Matthew, below are some questions, organized so you can respond more easily. --RobertC 17:03, 5 Feb 2006 (UTC)

• Can you run this yourself or does Gwicke have to do it? --RobertC 17:03, 5 Feb 2006 (UTC)

I can do it myself. The first time I ran it, it required a lot of manual work in excel. The second (and last) time I automated it a lot more so most of the hard part is done in SQL. I still did the formatting in Excel simply because it is easier for me to do that than writing it out in SQL. At some point though I will probably put it all in SQL so that I don't have to move data back and forth. Then I could just run the script and presto, the page would be updated. If I really got this right I could just ask Gabriel to schedule the script to be run. I don't think we are ready for that yet. Something like that really should have some other stuff too--like an alert if it fails. --Matthew Faulconer 06:03, 6 Feb 2006 (UTC)

• Have you ever tried querying the recent changes page? What are your thoughts about the updating approach I mentioned here, as opposed to the build from scratch approach your queries do here? If you can't reasonably implement an automated SQL query of the recent changes page, it seems updating by hand (by which I mean copying to excel and then running some macros or VB code) might be less work than the current amount of clean up you have to do after the current SQL commands are run--but I'm just guessing.... --RobertC 17:03, 5 Feb 2006 (UTC)

I'm not really sure what it means to query the recent changes page. The page feeds off a table in the background. I could query the table. One could could build a program to actually read the recent changes page and query it, but in a case where we have direct access to the db that seems like it would be more trouble than it is worth. Either way though, table or page, the problem is that the recent changes page (and the table behind it) only include the most recent changes. That means that if you didn't run this updating process frequently enough you would lose changes. This sort of differential processing just introduces a lot more stuff to worry about: when was the last update, do I care about deletes, how do I avoid adding duplicates, etc. In my experience a process that figures out everything from scratch is easier on the programmer and harder on the computer than one that just handles the updates--some sort of differential processing. At least in this case, where accessing the db is relatively easy but changing the program is relatively hard, I think that is true. If we were talking about a manual process then I would agree that differential processing would make sense. --Matthew Faulconer 06:03, 6 Feb 2006 (UTC)

Good points, you've clearly thought about this more carefully than I have (esp. regarding issues like how to deal with deleted pages etc.). --RobertC 15:56, 6 Feb 2006 (UTC)

• What are your thoughts about organizing this page better? I'd be inclined to make a separate page for each standard work, and then giving the option of transcluding them all together, but then maybe my obsession with transclusions is silly. --RobertC 17:03, 5 Feb 2006 (UTC)

I can see the value in having each of the standard works separated onto separate pages. I don't see it as a big value at this point. But I agree it would be nicer. This would require some programming changes and bit more manual work (for now anyway) but nothing huge. I'm not sure though, once you did that, what the point of transcluding them all together would be--though it wouldn't hurt either. --Matthew Faulconer 06:03, 6 Feb 2006 (UTC)

I agree, probably not a big value at this point. It sounds like this process can theoretically be entirely automated. Not a huge priority, but it helps in thinking about the possibilities of the site (by the way, I see the value of this wiki site growing exponentially with time; for now, blogs are exciting and the Mormon blog world in only a couple years old, which makes reading through blog posts on a particular subject, and reading through Sunday school lesson sites, not too unwieldy of a project; but with time, this will become more unwieldy and having a wiki site with better organization and such flexible updating and organizing capabilities wll become increasingly valuable).

Is a table of "pages with content" built when you run this code? Would it be possible (and if so, how hard would it be) to write a user interface that, for example, allowed a user to specify a group of chapters and then pull up a transcluded page of all the non-empty commentary pages (and talk page links) for those chapters? I'm asking from a long-term perspective (years, not months or weeks), thinking about what kind of options would help overcome the problem of having sparse pages and the limitation/inflexibility imposed by the 5-verse page system which we've adopted (which I still think is the best approach, I'm just thinking about the weaknesses of this approach and how to overcome them). --RobertC 15:56, 6 Feb 2006 (UTC)