http://qs321.pair.com?node_id=375925

All,
To be honest, I have only done a cursory investigation to determine if this has previously been discussed. I asked monks present in the CB and consulted Super Search with a basic query and think this may indeed be new.

When you sumbit a query to Super Search, it appropriately stops searching periodically to prevent too much strain on the servers. When you click "next" to continue searching, the current results are replaced by the results of the next time slice. I think it would be useful to be able to have a complete listing of results. I do not have a preference on the implementation but have a couple of possible ideas.

I freely admit I have no idea how much work is involved, how many more important things are already in the queue, or if I am the only one who thinks this will be useful.

You seldom get what you don't ask for.

Cheers - L~R

  • Comment on Getting Complete Listing Of [Super Search] Results

Replies are listed 'Best First'.
Re: Getting Complete Listing Of [Super Search] Results
by Theo (Priest) on Jul 20, 2004 at 14:24 UTC
    I also have seen a need for this. And as long as we're in the "if you don't ask, you never get" mode, along with the individual check boxes, I'd like to see a "check all" and a "clear all" pair of buttons.

    If that is feasable, could it be included on the newest nodes page, also? There are times when I don't have time to go through all of the newest nodes and they start to pile up. If I could check off the ones I'd read, there would be a shorter list for next time.

    -Theo-
    (so many nodes and so little time ... )

      Theo, theres a checkbox at the bottom of Newest Nodes which says "I've read all these" (or similar), maybe you overlooked it? As soon as you hit that once, however, NN will always show you all nodes since you last used it, instead of just the nodes for the last day, which it defaults to otherwise. If you'd like to turn it off again, theres a "Clear newest nodes flag" (or something), on user settings.

      C.

        Thank you, castaway. I use that button every day that I finish all of the NN. It's just that some days I get behind and it takes several days to catch up during which I can't hit that button or I'll lose the nodes I haven't seen yet.

        -Theo-
        (so many nodes and so little time ... )

Re: Getting Complete Listing Of [Super Search] Results (to-do)
by tye (Sage) on Jul 20, 2004 at 20:33 UTC

    Something like this has been on my to-do list for a long time. One of the big problems was interface design. But that is firming up in my head.

    FYI, there would be little impact on server load. The list of matching node IDs would go into a hidden field in the form. There would be a limit on how many matching nodes you could collect. You'd still have to hit "Next" to search the next chunk of the DB.

    Note that I don't find much need for this when using Super Search. I'm almost always searching within a specific section, for a specific author, or within a fairly small date range, etc. So most of my uses of Super Search have searched as far as I want them to after the first 'submit'.

    On the occasions when I have to hit 'Next', I've usually reviewed the nodes found (if any) and perhaps popped the most promising of them into separate windows/tabs.

    But being able to carry the list of selected nodes around would be handy and reasonable to implement.

    I like having a check box for each match and having a radio button for "keep only" vs. "keep all but" with the default being to "keep all but" (when you hit "Next"). And, when hitting "New Search", the option to "discard previous matches" (default), "add to previous matches ('or')", or "refine only previous matches ('and')" (if MySQL can do this efficiently enough that I don't need to worry about doing it in batches).

    Each time you hit "Next", I would only show the new finds and a count of previous finds and have a button to bring up the entire list.

    - tye        

      Im curious if you see a place for providing an XML version of Super Search?


      ---
      demerphq

        First they ignore you, then they laugh at you, then they fight you, then you win.
        -- Gandhi


        If that "you" was plural, Yes.

        A big part of the design of Super Search is that it requires interaction to complete a search that would take too long (because it was vaguely specified and/or because the DB server is busy). This encourages people (who are naturally at least a little impatient) to either refine their query or retry at a different time (or use something other than Super Search).

        An XML interface would either remove this `interactivity or likely automate it which removes the value of it.

        So an XML interface to Super Search sounds like something that would be a source of problematic load. But that is just a first reaction (I haven't thought about it much and I have to run ATM). I'll contemplate further on the subject.

        - tye        

Re: Getting Complete Listing Of [Super Search] Results
by pg (Canon) on Jul 20, 2004 at 16:28 UTC

    Hm... probably not. Potentially, this could easily expose the server to resource exhaustion attack. In order to support this, most likely the server has to remember the search result so far in memory (or somewhere), until the logical end is reached. This is costly, and could be used for attacking.

    Consider this design difficulty: the server cannot guess whether you want to continue search, so it has to remember everything as long as there is still possible results left unreported. An obvious question would be when should the server release the resources, and forget your historical result. This is almost an unresolvable question. There is simply no way the system can safely guess your itention, maybe use some sort of timeout, which is not ideal.

    The current way is nearly the best. (I am not a developer of this site, just a user ;-)

      pg,
      In this case I would change save/discard these to "add these node ids to my scratch pad/personal node.

      Cheers - L~R

Re: Getting Complete Listing Of [Super Search] Results
by ambrus (Abbot) on Jul 20, 2004 at 16:20 UTC

    I don't think this would be a good idea.

    1. How do you think that could be implemented? Would the previous store results (potentially a lot) be stored at the server and automatically discarded after some time? Or would it be reposted every time you click on next? Or else would Super-Search redo the search from the start every time? I think all of these are bad solutions, maybe there is a better way?

    2. Also I guess that it would increase the load Super-Search puts on the server, as many people would just keep clicking Next until the whole database is searched and only look at the results after that; so, if you understand what I mean, it would cancel the effect of Super-Search interrupting itself periodically.

      INAE, but could you not persist the connection, sleeping in between batches and wrapping each batch in a table.

      Does a mostly sleeping, persistant connection unduly tax Apache?

      Don't most (many, all) browsers render a table as soon as it's complete?

      The format of the links wouldn't have to change. Just put <table><tr><td> at the top of each batch and close it at the bottom, before sleeping for a few seconds then looping back to do the next batch.

      ...heading back to Barcelona.


      Examine what is said, not who speaks.
      "Efficiency is intelligent laziness." -David Dunham
      "Think for yourself!" - Abigail
      "Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon
      ambrus,
      How do you think that could be implemented?
      Given that I am not a devil - I have no idea how it would happen under the covers. I also wasn't too biased on the outside interface. You do bring up a good point:

      as many people would just keep clicking Next until the whole database is searched
      Again, I do not care how the functionality is acheived - even if it is nothing more than "add these node ids to my scratch pad/personal nodelet". I Just have found myself searching multiple times after discovering something I thought was not-important was later important.

      So - if there is a way to achieve this functionality without the potential resource abuse - I would like to see it added.

      Cheers - L~R

      as many people would just keep clicking Next until the whole database is searched and only look at the results after that

      I don't think that would be an issue. If the the default was for the articles to be unchecked and there was no select all mechanism, then you'd have to manually check all the ones you wanted to keep visible during your search.

      --
      "A long habit of not thinking a thing wrong, gives it a superficial appearance of being right." -- Thomas Paine
      naChoZ

Re: Getting Complete Listing Of [Super Search] Results
by idsfa (Vicar) on Jul 20, 2004 at 19:02 UTC
      idsfa,
      I use tab browsing and that is what I am doing currently. But depending on your search it can yield hundreds of matches. Hundreds of tabs suck.

      Cheers - L~R

        Hundreds of tabs sucks, but you wouldn't want to read through the results of a search that return hundreds of results, do you? If I write such a query, I either try to make the query more specific or else read only the first few pages.

Re: Getting Complete Listing Of [Super Search] Results
by kelan (Deacon) on Jul 21, 2004 at 14:41 UTC

    Note that you can do this right now (with a little finagling) with Opera:

    1. Open Super Search and start your search to get the first page of results.
    2. Right click anywhere on the page and select the menu option "Create linked window". (I'm not sure what the exact wording is, since I haven't used Opera in a while.) This will create a new window which will receive the results of anything you do in the current window.
    3. Hit the Next button on your search. The next page of results will appear in your newly created "linked" window, instead of replacing the page in your current window.
    4. Go to step 2 to create a window linked to your second page of results, which will display the third page, etc.

    Of course, you'll end up having N number of windows open, each with one section of the results. On the other hand, you can do it right now, before it actually gets implemented on the site.