User Poll: What do you want done with the Newsgroup data?

[MarkCulp]

Sybase iAnywhere believes that a Q&A forum similar to this one is the best solution for our customers to get answers to their questions about the SQL Anywhere product. As such, we are planning to move this SQLA forum to a new site powered by OSQA and SQL Anywhere hosted on sybase.com. We have been actively working on making software enhancements to the Q&A software that we feel are necessary before we can make the switch. Our goal is to make the new SQLA forum have all of the advantages of a modern question-and-answer HTTP web site but keep as many of the traditional advantages of NNTP as possible.

Once this new SQLA site is running, we are considering deprecating and eventually closing down the current NNTP forums that are related to SQL Anywhere. If this path is taken we would like to know from you, our user community, what you would like to see happen to the newsgroup articles and threads.

There are three basic options:

• Do Nothing: Leave the newsgroup forums as is, making them read-only so no new content is added.

• Searchable Archive: Move all of the newsgroup data to a web site so that it can be searched and viewed using a browser.

• Migrate to SQLA: Move the existing newsgroup articles into SQLA

There are several variations of the Migrate to SQLA option:

• Bulk Load all of the newsgroup articles into SQLA at one time

• Trickle Load all of the newsgroup articles into SQLA a few threads per day.

• Selective Load some of the newsgroup threads into SQLA.

• Selective Trickle load some of the newsgroup threads into SQLA a few threads per day.

For all of the Migrate to SQLA options, here is the common basic idea:

• Each newsgroup thread would become a separate question-answer sequence within SQLA with the initial newsgroup posting becoming the question and all responses becoming comments on the question.

• Comments would retain their same hierarchical structure that existed in the newsgroup forum.

• SQLA users would be able to edit the questions and comments to clean up the text (if they chose to do so) thus enhancing the overall content of SQLA.

• We are also considering providing a way for a user to "promote" a comment to be an answer so that users can mark which response in the newsgroup contained the answer to the original posting.

I have given more details for each of these options as separate individual answers. Please vote up the option(s) that you think would best serve you and the SQL Anywhere community, or add your own answer / option.

[VolkerBarth]

I would invite all voters to leave a short comment on their vote - just to increase the understanding of the pros and cons of each suggestion.

[VolkerBarth]

@Mark: For all those "Migrate to SQLA" options: Will the original NNTP contents still be available (in NNTP format and as read only), too? IMHO, that would be a prerequisite to "feel free" to edit/delete imported threads - I would not like to change/delete contents from others when the original contents would be unavailable.

[MarkCulp]

The way that (new) SQLA works is that all revisions of questions and answers can be viewed (I don't see this capability in the current SQLA?). Only the owner of the question/comment/answer or users with sufficient karma (aka reputation) can edit the content. If we migrate the NNTP data into SQLA we will need to consider if we want to make the enhancement to enable viewing of revisions of a comment.

[VolkerBarth]

@Mark: My concern deals more with the question if ex-NNTP contents could become unavailable just because someone (or a import filter) has decided is is not relevant anymore. - While I strongly agree that imported NNTP contents should be editable and deletable here, I might also rely on, say, this old thread from 2002 with a v5.5/SQL Remote question being still available somewhere for unforeseen maintenance cases.

[VolkerBarth]

@Mark: As the current trend here seems to focus on "Buld Load" vs. "Searchable Archive": When regarding the "Buld Load" way: Is there a need for a further poll w.r.t. "Must original NNTP contents be made available as-is, too?" (preferably in a searchable archive) - i.e. will someone miss contents when it is imported in SQLA but discarded as being irrelevant afterwards? - That is a point that seems mssing in the discussion so far.

[MarkCulp]

Migrate to SQLA (Bulk Load): Move all of the newsgroup articles into SQLA making each thread a separate question.

This solution would copy all of the content from the newsgroup forums into SQLA and hence make all of newsgroup information immediately available. This can be both a good and bad thing. It will be good because the information will easily be found in searches on SQLA but it is also bad because the newsgroup data would initially overwhelm the existing SQLA content and not all of the threads in the newsgroups contain any real useful information.

Getting all of the newsgroup data into SQLA in a format that would be suitable for a Q&A forum would not be a trivial amount of work. I have already been working on a process to do this with the realization that the end result will not be perfect. SQLA users would likely need to do further editing of the newsgroup content before it would be 100% acceptable. This effort would not have to be done immediately. SQLA users would be encouraged to clean up the historical newsgroup articles when they could.

[Former Member]

Could the responses to the original question be answers and not comments?

[MarkCulp]

We could make the NNTP responses to the original posting answers in SQLA but by doing so we would lose the hierarchy of the responses. Note that comments in the new SQLA site will be hierarchical in the same way that articles in the newsgroups are hierarchical. If we migrate the NNTP data to SQLA then we would also consider allowing a user to "promote" a comment to be an answer ... and all comments that were responses to the comment-that-is-now-an-answer would become comments to the new answer. Note also that comments in the new site will allow formatting.

[Former Member]

This would be the best option, because all information is together on one place. Otherwise I would have to do a web search, and then a SQLA search too.

[former_SQLA_member1694868]

I would also rather have one destination going forward.

[Former Member]

I think complete, one-stop migration is the way to go.

[VolkerBarth]

Reason for my downvote: Days ago, I have thought of this option as the best one. But after studying several newsgroups, I feel the contents of most NNTP threads would appear as "noise" in contrast to the current Q/As. Just think of all those "I cannot connect" or "My database is not starting" threads - must be thousands of them with usually similar answers. This site has not such questions so far - of course, it might change when it becomes THE place to ask those questions, too. - So I fear that too much manual afford would be needed to tell the pearls from the junk.

[Breck_Carter]

It's time to stop over-thinking this. No, there is not too much noise on the NNTP newsgroups, just like there are not too many books in the store, or too many movies in the theater, or too many programs on television. Yes, the "new questions" display will be useless for a few days after the bulk load, but it will quickly become useful again. Yes, some folks will keenly miss NNTP, especially those who have not used SQLA at all... it's up to the rest of us to welcome them. I vote "big bang" bulk load, and I pray that Google Search will make sense of it all.

[thomas_duemesnil]

I vote for this one. But when a users stumbles over junk it would be great if we could mark it for a removal review. If two or more users mark a message it could be hidden from normal view.

Please mark all imported Questions as answered and Accepted otherwise they will automagically pop up on the acive questions tab.

[Former Member]

Agreed-- I was initially all for the trickle load, but the burden of deciding what's useful is probably wasted time if we can just downvote or mark entries as junk as we find them. Also I wonder if it's possible in some way to bulk load everything that has at least two posts-- so that the one-post junk posts and unanswered, useless questions don't have to be ferreted out by the community.

[Former Member]

@TDuemesnil - I believe they should be left unanswered. Granted, it will make the 'active questions' list enormous, but just by importing there can be no guarantee that the question was actually answered (correctly).

[Former Member]

I'd like to have a single canonical resource where I can search for the community's collective knowledge. Even if I had to do without the community for a day or two during the transition and bulk load, I'd prefer that over weeks or months of having to look in two different places for an answer.

[reimer_pods]

I don't thing it would any good to move all NNTP contents to SQLA. Large parts of it are related to archived versions; only a small fraction of those messages would still apply to current versions.

I suppose that for me it wouldn't take long to make SQLA the resource of choice and disregard the rest.

[MarkCulp]

Searchable Archive: Move all of the newsgroup data to a web site so that it can be searched and viewed using a browser.

This solution is fairly easy to implement (and in fact we have been using a prototype of this solution internally at Sybase iAnywhere for quite a while). Users would be easily able to search the news articles to find relevant information.

If this solution is chosen, we would also provide a link from the SQLA site to the web site that contained the newsgroup articles and we would promote the newsgroup article site in the SQLA documentation and FAQ, etc. Users that found useful threads in the newsgroup data would be encouraged to create a new question in SQLA (perhaps a copy of the original posting in newsgroup) and provide an answer which would contain a link to the corresponding newsgroup response that answers the question (as well as optionally copying the text of that answer). Following this procedure would, over time, copy the best and most useful information from the historical newsgroup into SQLA.

[regdomaratzki]

This would be the best option IMHO. It promotes the new site since it's the only way to ask new questions, but makes available the 15 years of data in what will soon be the read-only newsgroups. I'm also under the assumption that the ability to search the newsgroups does not rely on Sybase continuing to host a news server. Bringing the newsgroup data into SQLA will result in a lot of poorly formatted data in SQLA unless a huge amount of effort is placed into importing the newsgroup data.

[reimer_pods]

Keep the contents available, but supply enhanced search abilities. That would be IT (IMHO).

[justin_willey]

I'd like this if somehow (ha ha) you could search within sqla, and then easily extend the search to the archive if you needed to., with results in a similarish format

[Breck_Carter]

Mark: Be brave, be confident, you know what's right, just do it!

Don't fall into the trap that destroyed the Sybase Developers Network, the trap of asking people what they want. People want teleportation and time travel, for free! Ivan might be able to deliver the technology but you don't have the budget.

The new SQLA will either work, or it won't.

But if you keep this discussion going, it definitely won't work.

[MarkCulp]

Migrate to SQLA (Selective Trickle) Move some of the newsgroup threads into SQLA a few threads per day.

This solution would be a combination of Trickle Load and Selective Load. A few selective newsgroup threads would be added to SQLA each day so that the regular activity of SQLA would not be overwhelmed by the newsgroup data and only threads with useful content would be copied.

Like Trickle Load, it would take several months before all of the selected threads were loaded into SQLA.

[former_SQLA_member1694868]

There are a few reasons I vote for this option. 1) For those of us that have not read the newsgroups, but have now found this site we can receive the "best" of the newsgroups as a few new posts a day. 2) Ultimately this site is the desired destination going forward, not a separate site. 3) Migrating a few at a time will allow the community time to digest and clean up each one, voting on an answer, and making any necessary edits, without placing the "cleanup" burden solely on the person migrating the data.

And yes, I voted twice.

[MarkCulp]

@Siger: Nothing wrong with voting more than once - this lets us know which options you will be happy with if we proceed with deprecating the newsgroups. Thanks for voicing your opinion.

[VolkerBarth]

I second Siger's arguments (except that I do use the newgroups:). Confine my comment on the "Bulk Load" option.

[Breck_Carter]

If you are looking for volunteers, count me OUT... too much work.

[Former Member]

Do nothing. Leave the newsgroups read only and do nothing.

Technology moves on and the effort required to move all history to largely unsupported product issues is largely pointless IMHO.

Rather than dedicate the effort to that, calculate how much effort that would involve and then keep more staff posting in the new format to ensure that people do not have difficulty: a) using the new format. b) finding answers in the short to medium term.

[Former Member]

Go back. I'd prefer staying with NNTP, because I've got a range of available clients, and I don't have to deal with some web site's preferred interface style. I realize this wasn't one of the offered options, but if I get to vote, I'd rather have NNTP. I guess this makes me one of the luddites ...

[VolkerBarth]

Just to add: What NNTP clients do you use to your contentment? - I have used both Outlook Express (formerly) and Thunderbird (now), and both have always had issues with wrongly assembled NNTP threads and the like (although more seldom with Thunderbird). Don't know if that is a NNTP problem or one of the clients, though...

[VolkerBarth]

Mark, as with any software project, I'm trying to ask "Can you show us a prototype?".

For all those "Migrate to SQLA" options (which is the general way to go IMHO), I guess the basic question is how much noise gets into the site when doing a bulk load.

I see the clear advantage when data is imported only once - you have to search in only one place, and it prevents a possibly nearly-ever-lasting import process.

But if the site would become quite useless because of all that stuff, the attraction of the site could go down badly.

So I would like to ask politely if test sites could be made available for the more popular options...

Just two more variants come to my mind:

1. For the three "Migrate to SQLA" options except "Bulk Load": Could there be an "import option" in SQLA to the NNTP (or "Searchable Archive") site? - I have done that a few times manually, and it works (and inherently invites to reformat the contents) but possibly that could be made "easier"/"smoother" for less experienced users or could have features to reassign the original user when they do exist in SQLA, too.

   That would be a way to accelerate a "selective" and/or "trickle" approach by letting the community add valuable threads on their own. (Obviously, it would require some kind of registration if a thread is already imported by the sytem/user or not.)

2. For all options except "Buld Load": I would suggest that a bunch of FAQs (e.g. connection problems, not starting servers and the like) that make up a huge amount of the NNTP topics would be added by the support staff before the new site becames the NNTP replacement. That way users with these questions could get a thorough answer even before they have to raise a question - again with the goal to prevent similar questions. I guess Jeff Albion, Josh Savill and their collegues have very valuable contents to share from their lots of NNTP answers.

[Former Member]

I don't think it's worthwhile, Volker, to implement the "selective" copy approach from NNTP to SQLA. It also assumes the continued existence of the NNTP servers, which is not a given.

As for the FAQ - a great idea, but it's not going to happen. It would be exceedingly time-consuming for a team to troll through 20 years of NNTP content to look for the "best" postings that should go into the FAQ - not only because there are tens of thousands of posts, but I can't imagine writing a piece of code to "judge" the quality of the responses.

[VolkerBarth]

@Glenn: Just to clarify my second point: I do not at all ask for a "quest for the best NNTP articles". I agree that this would be pointless. My suggestion was just to ask those very experienced supporters to add a few articles for those questions that get asked a lot (and that they are used to answer). Should not be more than a few hours work, methinks. - Possibly that could be similar to the new FAQ page in SA 12 (which is a great IMHO), cf. http://dcx.sybase.com/1200en/saintro/fg-faq.html.

[MarkCulp]

Migrate to SQLA (Selective Load) Move some of the newsgroup threads into SQLA all at once.

This solution would copy only a selective set of newsgroup threads into SQLA - the criteria on which threads were copied would be open to discussion. Like the Bulk Load solution, the selected threads would be added to SQLA all at once. The goal of the selection criteria would be to remove threads that contains little to no useful content.

[VolkerBarth]

What kind of criteria are you thinking of (just as an example)?

[MarkCulp]

The exact criteria is to be determined, but certainly there are a lot of articles/threads in the newsgroups that do not contain relevant data. For example, duplicate articles (e.g. cross postings and questions that are duplicates of a previous question - e.g. "How do I connect to a database?"), postings that never had any responses, various assortment of announcement-type postings that are no longer relevant, article threads that have "errors" (missing articles including ones that the original posting is missing), etc. We would need further discussion to determine the criteria.

[Breck_Carter]

Too much effort, too little value... nobody is going to read the material like a book, the noise does not matter.

[MarkCulp]

Migrate to SQLA (Trickle Load) Move all of the newsgroup articles into SQLA a few threads per day.

Like the Bulk Load solution, eventually all of the newsgroup articles would be copied into SQLA. The advantage of copying only a few threads per day into SQLA would be that the "new questions" and the "active questions" list would not be overwhelmed with the real new questions and real active questions. In addition, users could chose to spend a few minutes each day to edit the historical newsgroup threads (that were added that day) to revise the content to conform to the Q&A format and as a result make the information more useful.

The disadvantage to a trickle load approach is that it would take several months (years?) before all of the newsgroup threads were loaded into SQLA. Some prioritization of the threads would likely be desirable so that the threads with the most useful information were loaded first.

[Breck_Carter]

This sounds like Plan B in case the bulk load brings the Sybase infrastructure to its knees. Let's be honest, most of sybase.com (including DCX) performs very badly, and has done so for many years.

[MarkCulp]

Do nothing: Leave the newsgroup forums as is, making them read-only so no new content can be added.

This is certainly the easiest thing to do. It would require that the news forum servers remain online so that users could still access them using an NTTP news reader. Eventually the servers would be turned off. The exact timing of their disappearance is unknown.

[VolkerBarth]

I think this is not useful as a) it has always been difficult to do searches in NNTP forums (both with web UIs and NNTP readers) and b) it is difficult to link to NNTP articles from here - at least, for me...