[Solved] How to fix broken 'Statistical prediction' (Thunderbird v91.x, Nostalgy++ v3.0.11) after migrating to new IMAP-service provider?

Is there some way I can fix/rebuild/etc my Thunderbid-Nostalgy++ Statistical prediction predictive performance?

Symptoms

I’ve used Nostalgy/Nostalgy++ for years/decades. Its Statistical prediction for “moving/copying emails to a different folder” has worked marvelously, until I recently upgraded to v91.4.1. (At least, that’s what I presume the Statistical prediction feature does; if wrong, pls advise.)

Now the only folder reference(s) that come up for an email move/copy is the last folder that was moved/copied to (via a Nostalgy++ move/copy).

More details

Important note: I switched IMAP servers, porting the email-folder data from the old to the new, deleting the old-imap-server’s Thunderbird-local-my-macOS account (named Personal), and creating a new account (also named Personal, purposely substituting for the old account with the new server and the exact same email-folder data).

Does the above maneuver mess things up for Nostalgy/Nostalgy++? If so, is there a way to recover/rebuild/export+import the previously-working-on-previous-IMAP-server/account Nostalgy data so that the Statistical prediction can be reused on the new account (that has the same email-folder data)?

I have years/decades of past moves/copies and therefore (theoretically) lots of stats data to draw from for Nostalgy’s Statistical prediction stuff. I also have several clean, historical copies of my Thunderbird data profile folder (which is where I presume Nostalgy keeps it’s prior stats data) saved, including but limited to prior to the Thunderbird upgrade (which includes v91.4.1 and v68.12.1 profile-folder saves).

If needed: is it possible to rebuild the Statistical prediction “database” from one of these past, “clean” Thunderbird profile folders?

My current versions: Thunderbird v91.4.1 Nostalgy++ v3.0.11 macOS 10.14.6, build 18G9323

[johnnyutahh]

I fixed this. And I was able to recover and reuse all the years of historical, predictive data I had built up (in Nostalgy’s nfpredict.sqlite database file) saving me tremendous effort of not having to manually recreate all the predictive-move data for hundreds (maybe thousands?) of email folders. (Woo hoo!)

If the following procedure/directions do not make total sense and you’d like me to clarify/fix these directions, pls signal me with a reply (on this issue) with the details of what’s not clear to you, and I’ll try to fix the text. If I can, I’ll turn this into a git-Asciidoctor-markup-pullable file so others can edit it (or just a wiki file/page someplace)… but no time to do that now.

The following procedure is more strictly-text-based and less “Github-issue-markup-text based,” because they (the procedure) was borrowed from a git commit (of my Thunderbird app-profile folder).

Anybody who wants to convert this into a proper markup (asciidoctor/markdown/etc) text file directions, preferably in a git repo, are more than welcome to do so. (I tried to use some markup-language stuff in the content to make this easier.) I’ll even review it for you. 😃

~Johnny

Details:

Reiterating: I switched IMAP-email-server/service providers AND upgraded
Thunderbird MUA clients versions close to the same time. (Technically
I did one before the other... but these changes were within a couple days
of each other.)

Specifically:

A. upgraded Thunderbird from 68.12.1 to 91.x, converting from Nostalgy
   to Nostalgy++ along the way.

B. changed 'Personal' email account from account@Provider1 to
   account@Provider2 (redacted) and migrated all the email+folder
   content (from Provider1 to Provider2) along the way.

Changing (A.) does not seems to impact anything. Details provided
at the end of this note.

Here's the procedure I followed to fix my Nostalgy++
email-move-to-predictive-target-folder feature to use the historical
'nfpredict.sqlite' database:

- I converted the Nostalgy/Nostalgy++ 'nfpredict.sqlite'
  predictive-email-folder database file to match with the
  pre-'Personal' email-service-provider change by converting
  'imap://[e-mail]' pathspec references to
  'imap://[e-mail]' pathspec references.

- Said 'nfpredict.sqlite' file "conversion" was performed by:

  [IMPORANT: I recommend taking extreme caution when updating the
  Thunderbird-app-profile folder. I do this by git-controlling
  said folder and issuing frequent 'git add -A .' and 'git commit'
  commands--only when Thunderbird is not running, of course--for every
  little change that I make to the Thunderbird-app-profile folder,
  including the 'nfpredict.sqlite' database-file change discussed
  below.]

  (some of the following `.sql` and `.sqlite.*` filenames are arbitrary)

  1. stop Thunderbird. sqlite3-dump the original ('nfpredict.sqlite')
     database file from the Thunderbird profile folder to csv/text:
     `$ sqlite3 nfpredict.sqlite.original .dump > before_conversion.sql`
  2. (paraphrasing)
     edit `before_conversion.sql` (from step #1 above) with:
     `sed s|imap://[e-mail]/|imap://[e-mail]/|`,
     or however you want to edit said .sql/csv/text file
     (I used a vim `:%s` command similar to the above `sed` command),
     producing a `converted.sql` file.
     note: The Provider2 pathspec references were determined by testing
           Nostalgy++ on the new Provider2 account with some email
           moves and looking at the resluting new-'imap://' pathspec
           database entries (in the dump/csv/text file)... and then
           using this resulting Provider2 pathspec on the conversion
           of the previous-to-Provider2 database dump/csv/text file
           to create the input file for the next (#3) step. Or... you
           might just be able to figure out the Provider2 pathspec by
           intelligently-guessing or some other way.
  3. importing `converted.sql` into a new 'nfpredict.sqlite' file:
     `$ sqlite3 nfpredict.sqlite.new < converted.sql`
  4. overwriting the 'nfpredict.sqlite' file in the Thunderbird
     profile folder with the `nfpredict.sqlite.new` file
     (created in step #3 above).
  5. start Thunderbird


My system's info/versions. I've no idea why `sqlite3 --version` number is
different than the `brew info sqlite3` number(s). :-/

  $ sw_vers
  ProductName:	Mac OS X
  ProductVersion:	10.14.6
  BuildVersion:	18G9323
  $ sqlite3 --version
  3.24.0 2018-06-04 14:10:15 95fbac39baaab1c3a84fdfc82ccb7f42398b2e92f18a2a57bce1d4a713cbaapl
  $ brew info sqlite3 | head -5
  sqlite: stable 3.37.2 [keg-only]
  Command-line interface for SQLite
  https://sqlite.org/index.html
  /usr/local/Cellar/sqlite/3.36.0 (11 files, 4.2MB)
    Poured from bottle on 2021-11-25 at 17:50:56
  $ which sqlite3
  /usr/bin/sqlite3
  $

"Changing (A.) does not seem to matter" details:

(A.) "Upgrading Thunderbird from 68.12.1 to 91.x" does not seem
to matter in this conversion/fix process, best I can tell.
Nostagly/Nostalgy++ does seem to (as expected) update its embedded
`sqlite3` database version over time/version_upgrades (I can tell
this by running `$ file nfpredict.sqlite`, which reports the
sqlite3-database-engine version that generated the file--assuming
`file(1)` on macOS homebrew is accurate), but that too does not
seem to make a difference in the process. It all seems to "just
work" irrespective of the versions. The sql schema does not
seem to change over the life of these versions. THANK YOU Nostalgy
project developers for not messing_with/changing the schema of the
predictive-move-folder-target database as you developed new vesions of
Nostalgy/Nostalgy++! :-) :-) :-) Otherwise, this whole procedure whould
have been much harder if not close to impossible.

[johnnyutahh]

@johnnyutahh Johnny, did you look into using a SQL UPDATE instead of sed? I haven’t dug into the schema yet myself - I’m thinking it would be easier to incorporate into the extension if a sqlite3 SQL statement would work.

I have not. But it seems like a SQL-based statement (and therefore an extension-feature addition) might work just fine; I have no reason to think that it would not.