Re^5: String Comparison & Equivalence Challenge

in reply to Re^4: String Comparison & Equivalence Challenge
in thread String Comparison & Equivalence Challenge

It turns out this text is not so interesting because there is so much 'formula', for want of a better word, and these large parts of 'formula'-sentence while meaning little will give the comparison a high hit number -- see below. By the way, this will be the same for the approach of Algorithm::Diff's LCS (the TK program that tybalt89 made you). I think, anyway.

Yesterday, I generated comparisons and kept all above 0.25. This produced a table with almost 40M comparisons with their 'similarity' number.

It 'worked', in a way, but the result is still a bit disappointing because of the type of text this is (I think). A more real information text with less repetition, less fluff, if you see what I mean, might be more interesting.

select
    substring(cmp.sim::text,1,5) sim
  ,    k1.text || chr(10)
    || k2.text || chr(10)

from   kjv_simil_0_25 cmp   -- big table
  join kjv k1 on id1=k1.id  -- table from the KJV file
  join kjv k2 on id2=k2.id  --   ,,

where k1.book = 50  -- look at just this book
  and sim > 0.7   -- remove too different
  and sim < 0.85  -- remove too identical
;

  sim  |                                                              
+?column?                                                             
+  
-------+--------------------------------------------------------------
+---------------------------------------------------------------------
+--
 0.833 | The grace of our Lord Jesus Christ be with you all. Amen.    
+                                                                     
+ +
       | Brethren the grace of our Lord Jesus Christ be with your spir
+it. Amen.                                                            
+ +
       | 
 0.833 | The grace of our Lord Jesus Christ be with you all. Amen.    
+                                                                     
+ +
       | The grace of our Lord Jesus Christ be with your spirit. Amen.
+                                                                     
+ +
       | 
 0.769 | Grace be unto you and peace from God our Father and from the 
+Lord Jesus Christ.                                                   
+ +
       | To Timothy my dearly beloved son: Grace mercy and peace from 
+God the Father and Christ Jesus our Lord.                            
+ +
       | 
 0.769 | Grace be unto you and peace from God our Father and from the 
+Lord Jesus Christ.                                                   
+ +
       | To Titus mine own son after the common faith: Grace mercy and
+ peace from God the Father and the Lord Jesus Christ our Saviour.    
+ +
       | 
 0.736 | Grace be unto you and peace from God our Father and from the 
+Lord Jesus Christ.                                                   
+ +
       | Grace be with you mercy and peace from God the Father and fro
+m the Lord Jesus Christ the Son of the Father in truth and love.     
+ +
       | 
 0.723 | Grace be unto you and peace from God our Father and from the 
+Lord Jesus Christ.                                                   
+ +
       | Unto Timothy my own son in the faith: Grace mercy and peace f
+rom God our Father and Jesus Christ our Lord.                        
+ +
       | 
 0.714 | The grace of our Lord Jesus Christ be with you all. Amen.    
+                                                                     
+ +
       | The Lord Jesus Christ be with thy spirit. Grace be with you. 
+Amen.                                                                
+ +
       | 
 0.702 | Now unto God and our Father be glory for ever and ever. Amen.
+                                                                     
+ +
       | Saying Amen: Blessing and glory and wisdom and thanksgiving a
+nd honour and power and might be unto our God for ever and ever. Amen
+.+
       | 
(8 rows)

Time: 16.557 ms
[download]

Comment on Re^5: String Comparison & Equivalence Challenge Download Code

Replies are listed 'Best First'.
Re^6: String Comparison & Equivalence Challenge by Polyglot (Chaplain) on Mar 15, 2021 at 13:40 UTC
You make some very worthwhile observations. From my perspective, the extensive similarities of the various texts is what provides the incentive to identify them. If the text were much more diverse, it would be difficult to find comparisons with a sufficient similarity index to pass the threshold, at which point the results would lose significance. Any algorithm that gives less weight to the more common words will, in my opinion, be less able to handle this particular corpus and yield quality results. It is true that many of the keywords are used ubiquitously throughout the Bible: words like Lord, God, father, son, praise, faith, etc. There are also words that are much more common than in other types of text, such as inherit, blessing, perpetual, covenant, sacrifice, etc. And then, as has been pointed out, many of the words are of old-English form and would not be easy to accommodate with a stemmer: verb forms like hadst, hast, hath, gavest, giveth, rebuketh, etc.; pronouns like thee, thou, thine, and ye; and words like hence, hither, whither, whence, lest, whatsoever, whomsoever, etc. To add a little spice, there are the more unusual vocabulary words, like apothecary, winnowing, and all the wonderful names like Belteshazzar, Nebuchadnezzar, Abaddon, Apollyon, Zaphnathpaaneah, Mahershalalhashbaz, Sennacherib, etc. There are some interesting features of the language, along with some of the more repetitious aspects. The entire corpus contains close to 13,000 words--not far from the average vocabulary of a high-school graduate these days. A list of all the words in the KJV Bible, showing the number of occurrences for each, can be found HERE. It helps one to understand the peculiarities of this particular lexicon. Blessings, ~Polyglot~	[reply]
Re^7: String Comparison & Equivalence Challenge by LanX (Saint) on Mar 15, 2021 at 13:58 UTC
> A list of all the words in the KJV Bible, showing the number of occurrences for each, can be found HERE. So what's the difference between `Ashchenaz` and `Ashkenaz` ? Cheers Rolf _{(addicted to the Perl Programming Language :) Wikisyntax for the Monastery}	[reply] [d/l] [select]
Re^8: String Comparison & Equivalence Challenge by erix (Prior) on Mar 15, 2021 at 14:15 UTC
`$ psql psql (14devel) testdb=# select strict_word_similarity('Ashchenaz', 'Ashkenaz'); strict_word_similarity ------------------------ 0.46153846 (1 row) Time: 1.381 ms` [download]	[reply] [d/l]
Re^9: String Comparison & Equivalence Challenge by LanX (Saint) on Mar 15, 2021 at 14:24 UTC
Re^8: String Comparison & Equivalence Challenge by Polyglot (Chaplain) on Mar 15, 2021 at 14:38 UTC
Good question. They were two different people, and, perhaps because of this, the Masoretes added a different set of vowels/pronunciation marks to each of the two in the original Hebrew. The actual Hebrew consonants between the two words are the same. I suppose it would be a little like Freddie vs. Freddy -- they're basically the same name, but represent two different people; or maybe Rebekah vs. Rebecca, etc. Blessings, ~Polyglot~	[reply]
Re^9: String Comparison & Equivalence Challenge by LanX (Saint) on Mar 15, 2021 at 14:49 UTC
Re^10: String Comparison & Equivalence Challenge by erix (Prior) on Mar 15, 2021 at 15:07 UTC
Some notes below your chosen depth have not been shown here
Re^10: String Comparison & Equivalence Challenge by Polyglot (Chaplain) on Mar 15, 2021 at 15:26 UTC
Some notes below your chosen depth have not been shown here

In Section Seekers of Perl Wisdom