* TMG sources not importing

[Pblackwood]

I am new to FH. I am trialling transferring some projects from TMG 9.5 to the trial version of FH 6.2

I have a number of separate projects, some of which have imported OK. But there are several for which no sources have come across, though other data they contain appears to come across OK.

I have tried the following, each without success:

1. Direct import
2. Converting to GEDCOM and then importing
3. Reinstalling TMG
4. Re-indexing, optimising and validating project before importing
5. As suggested in the Knowledge Base, creating a blank project in TMG, then merging with the project to be imported.

These projects have one thing in common. Over the years they have each been started by copying an existing project and saving with a new name. The project they all ultimately originate from has imported OK, including sources.

The original project and all subsequent ones have only ever been compiled in TMG.

Finally, I have tried creating a new project from scratch in TMG and the sources for this transferred OK.

It appears that there is something amiss with just this set of projects.

Any ideas appreciated.

Peter

[Jane]

It does sound like the problem is in TMG as if it's not finding the sources when exporting a gedcom it suggests they are not linked to the facts. If you view the projects within TMG do the sources show against the correct items with in it?

[tatewise]

Welcome to the FHUG Peter.

As Jane says, first double check that the Sources are correctly linked in TMG.

I presume you are saying that in the Records Window on the Sources tab there are no records listed at all?

I believe TMG has a Project structure that may involve multiple Data Sets.
I am not sure whether the process of exporting/importing operates at the Project or Data Set level.
Maybe Sources are somehow shared by Data Sets in the same Project and that is where the problem lies.
Perhaps more expert TMG migrants can provide a more comprehensive answer.

FH Projects are less sophisticated, so it may be better to merge Data Sets in TMG prior to migrating to FH.

[Pblackwood]

Thanks Jane and Mike for your thoughts

If you view the projects within TMG do the sources show against the correct items with in it?

Yes, all appears well within TMG. This is an active database to which I am still adding data and sources.

...I presume you are saying that in the Records Window on the Sources tab there are no records listed at all?

That’s correct. And when I run a query “Sources -all”, it returns a blank report.

I believe TMG has a Project structure that may involve multiple Data Sets.
I am not sure whether the process of exporting/importing operates at the Project or Data Set level.
Maybe Sources are somehow shared by Data Sets in the same Project and that is where the problem lies.
Perhaps more expert TMG migrants can provide a more comprehensive answer.

FH Projects are less sophisticated, so it may be better to merge Data Sets in TMG prior to migrating to FH.

I only use a single dataset per project. Having said that, however, the knowledge base suggestion of merging a blank project does result in the creation of a second dataset in the original project. This can then be merged within the project to return it to a single data set comprising the combination of the original and newly imported (merged) data.

I tried importing the merged project both before merging the datasets (that is, with the original project dataset and the newly merged project dataset as two datasets in the import project) and after merging the datasets (that is, after merging the original project dataset and the newly merged project dataset into a single dataset in the import file). It made no difference to the outcome - in both cases no sources came across cross!

I will now try merging in TMG a project which does import OK with one that does not, and see if that helps to isolate the problem. In TMG projects can be merged in two directions, Project A into Project B, or Project B into Project A. It’s just possible these might result in different outcomes wrt whatever is the cause of this issue.

Thanks again for your assistance

Peter

[tatewise]

Peter, your symptoms are quite unusual, as I don't recall missing Source records being reported before.

Are there any significant exceptions reported in the import log that might give any clues?
In FH see Window > Log Files.

Have you tried Tools > Uncategorised Data > Find Uncategorised Data as advised in how_to:handling_unrecognised_data_fields|> Handling Uncategorised Data Fields (UDF) but bear in mind that Plugins are the one feature NOT supported in the trial version of FH.

I suggest you report the problem to Calico Pie using the how_to:about#problem_reporting|> Problem Reporting guide.
They may wish to see a copy of your TMG project files to investigate.

[Jane]

As TMG is not exporting Sources to a Gedcom either, this suggests it's within TMG that the problem lies. Can you confirm when you export a Gedcom file if you search the gedcom in a text editor there are no SOUR lines?

[Pblackwood]

Thanks for the suggestions. Will follow them up in the next few days and report back.

Peter

[DonF]

Peter,
I find your experiences here somewhat surprising, but if you could supply more detailed info it may help lead to a solution.
(1) if you do a GEDCOM export to transfer to fh, in the TMG Export wizard, step 5, you MUST tick 'Sources' (and selecting the 'Enhanced GEDCOM export' is also highly recommended). But I would never use GEDCOM as a transfer method to fh - the direct import is much better.
(2) so, for a TMG project that has this 'dropped Source' issue, can you supply the information from Step 2 and step 12 of the direct import? Step 2 states how many Sources fh is importing and Step 12 states how many Citations it finds. If either of these are 0, then I think we have a strange problem to follow up. If they're non-zero, then clearly fh has found SOMETHING.

I'm unaware of any log that fh creates during the import process, so if anybody has any knowledge of one, that may also help to pinpoint what is happening.

[tatewise]

Don, log files are not produced for direct import, but they are for GEDCOM import, and saved in Window > Log Files.

I believe Peter knows that direct import is best, but was using GEDCOM as a diagnostic aid.

[Pblackwood]

Window>Log Files shows no log file for either the direct and GEDCOM imports.

UDF report found 10 items, but none relate to sources.

But here's something interesting. In the FH Records Window, no sources are listed. However, the repository records, which are attached to each source record in TMG, have come across (over 100 records). Likewise the media records, most of which in TMG I have attached to source records, have also come across (over 900 records).

Still working on it!!

[Pblackwood]

Following Don's suggestion:

It skips Step 2! An indication that the sources are definitely not coming across? But repositories do (Step 1, 109 records) and citations at Step 12, (86070 records read).

Peter

[DonF]

OK,now we're getting somewhere!
For one of your 'suss' TMG projects try this:
Test #1: do a TMG Export to GEDCOM (make sure you have Sources ticked in the export wizard, step 5) and either look at the result, or import it into fh. In either the GEDCOM or fh, look for SOUR records..
Test #2: go to the folder where your TMG projects reside. As you probably know a TMG 'project' consists of the 'header' file with a filename of 'projectname.PJC' plus 60-80+ other files which are the actual database. For each database table type, there are separate files. For sources, you will have 4 significant files, namely 'projectname_M.DBF', 'projectname_A.DBF', 'projectname_S.DBF', 'projectname_U.DBF'. The M file contains the Sources you have defined, the A file is the Source types, the S file is the Citations and the U file the Source Elements used. Feel free to examine these with Notepad and you'll get a feel for their contents.

Now if any of these 4 files are missing, you have a significant TMG problem (but given all the validate etc tasks run I don't expect that to be true). So if they're all there and validation runs OK, you have a fh problem. Given it's finding citations, the S file must be recognised, but I have no idea what fh is doing with the records, if there are no sources. Finding Repositories before Sources makes sense as it would need Repositories before processing the Sources that point to them - which also means it's found the 'xxx_R.DBF' file.
Have a look and report back?

[Pblackwood]

Thanks Don,

Test 1 results.
I tested the following:
Original TMG project to GEDCOM.

Then:

GEDCOM back to TMG. Sources imported into new TMG OK
New TMG to FH. Sources imported (hooray!|).
GEDCOM to FH Sources imported (hooray!|).

But,
Original TMG direct to FH. Still no sources importing!

Test 2 results.
The A, M, S and U DBF files in the TMG project are all present and populated. For example, the M files contains all the 1800 + sources in the project.

My conclusions:

The TMG project files seem to be OK.

If I transfer via GEDCOM, the sources come across (though when I first tried this a few days ago they didn't - perhaps some error on my part). The TMG to GEDCOM threw up over 100 pages of "warnings" relating to unknown and expected Level 1, 2 and 3 Tags. I don't know what that's all about. I may have to pay more attention to the export options.

There remains a problem with directly importing the original TMG project!

Peter

[Jane]

Peter, it might be worth asking Calico to look at your TMG project, obviously they would need to have a copy of your data, there is obviously something "different" about your specific project set up which is causing the problem.

[Pblackwood]

Yes, I agree. That will be my next step.

Peter

[tatewise]

If you remember, that is one option I suggested on Monday:
"I suggest you report the problem to Calico Pie using the how_to:about#problem_reporting|> Problem Reporting guide.
They may wish to see a copy of your TMG project files to investigate."

At least you've discovered the GEDCOM includes Source records now, and thus focusses attention on the FH direct import.

[Pblackwood]

Thanks Mike, Jane and Don for your suggestions. I wanted to satisfy myself that the probability of the problem being either at the TMG end, or a user error, was low, before referring to support. Also, I have now actually purchased the program, so feel a bit more comfortable about calling upon support. I'll report back on the outcome.

Peter

[DonF]

It does sound like some sort of fh bug, though I have no idea why your project(s) should be so subtly different.
As has been suggested it would seem Calico Pie is the next port of call. I'm sure from their point of view that if you can reproduce the problem in a small TMG project (say no more than a dozen people) that would make debugging easier for them.

On the 100 or so pages of issues that TMG threw up during a GEDCOM export, I imagine most would be due to TMG warning you that GEDCOM can't handle the data TMG is trying to transfer. The export is complicated, by the fact that TMG has far more Source Elements than there are different tag types for sources in the GEDCOM specification. Split memos are another complication too, which i doubt that even the fh direct transfer can handle.

There is a very good explanation of what TMG tries to do at https://tmg.reigelridge.com/Sources-exporting.htm - just don't expect the results of a transfer to be as coherent as the original in TMG.

[Pblackwood]

FH Support found there were a number of source entries in the TMG _m file which were missing a value for 'type'. Once these were identified and rectified (fortunately there weren't too many), the problem was solved. I am very impressed with the service I received from Support. Thanks equally for all your suggestions, which while not solving the problem, helped me narrow it down before referring it to Support. Much appreciated.

Peter