Using bugzilla import script with older versions of Bugzilla (2.20)

Hello Selezen,

Is it possible for you to provide us with your Bugzilla database backup? We'll fix the issue and provide you with the complete imported YouTrack database.

Thank you.

That's a very generous offer!

However, there are two issues that you should be aware of:
1: the database backup is 1.36GB in size.
2: the import would need to append an existing YouTrack database.

Part of my job has been to investigate whether the import process would merge the data or replace the existing YT content.

Also, the database contains sensitive business information that my management would be reluctant to release without consultation, so I would need to obtain permission first, and an agreement that the data would not be retained by yourselves.

If you can answer the two issues above, then I will look into the possibility of releasing the file.

Thanks for your offer and your help. :)

Selezen,

Since the process of approving NDA takes long time period, we can test this script with Bugzilla 2.x on our side and send it to you.
If it doesn't fix your problem , we'll fix it stright on your database.

Database size is OK.
Import would append an existing YouTrack database.

I beleive, tomorrow we'll be able to provide you with the updated script.

Thank you.

Andrey,

That is fantastic - thanks for your help. I look forward to hearing from you tomorrow. If there's anything else I can provide you with then please let me know.

Hi Andrey - any progress on this?

Hello Selezen,

Sorry for delay. It's actualy in progress yet. I will let you know when it's ready asap.

Thank you.

Hi again, Andrey.

Sorry to push on this, but my manager is asking if there is an estimate on how much longer the script will take to put together. We have a deadline on this migration that is edging closer and he's starting to get concerned.

Thanks again for your work.

Dave

:)

EDIT: I should add that we are evaluating the software at the moment - our test licence expires a week today and we want to get the evaluation done before then - hence the urgency. :)

Hello Dave,

Sorry for that inconvenience with estimations. We've met several issues while customizing import script with Bugzilla 2.x., that should be fixed. For now, I would like to recommend you to upgrade your current Bugzilla to 3.x version and use our current import script.
If it's not suitable for you, please wait for 2.x script. We plan to release it till Monday.

Once again sorry for that inconvenience.

Hi andrey, and thanks for getting back to me.

We already tried updating the Bugzilla database / installation and we met several issues ourselves, hence why I opted to ask if there was a Bugzilla 2 script.

We'll wait for your script on Monday if that's OK. Thanks.

Dave

Hello Dave,

Please, download latest library https://github.com/JetBrains/youtrack-rest-python-library/ . In 'python/bugzilla/defaultBzMapping.py' you'll see detailed instruction in comments of how should you run this the script in comparison of Bugzilla 3.x versions.

Thank you and let us know if it helps.

Hi Andrey! Thanks for doing this!

It seemed to be going well, then an error message popped up:

File "c:\python\lib\site-packages\MySQLdb\connections.py", line 36, in default errorhandler raise errorclass, errorvalue
mysql_exceptions.ProgrammingError: (1146, "Table 'bugs.attach_data' doesn't exist")

Dave

Dave,

Did you execute queries before import as it's written in comments?

Yes I did - followed all the instructions there.

It would seem that it's looking for an "attach_data" table. I think the attach_data table is present in Bugzilla 3 but the equivalent in BZ2 is the "attachments" table.

EDIT: apologies, on checking the two databases it seems that the BZ3 database also has an "attachments" database - the "attach_data" table is unique to BZ3.

Dave

Dave,

We've fixed this. Please, download the latest library and try once again.

Sorry for the inconvenience.

Thanks again for your quick work!

That version seemed to be working well until getting this message:

Traceback (most recent call last):
  File "python/bugzilla2youtrack.py", line 350, in <module>
    main()
  File "python/bugzilla2youtrack.py", line 31, in main
    bz_product_names, lambda issue: True)
  File "python/bugzilla2youtrack.py", line 320, in bugzilla2youtrack
    , created=str(int(attach.created) * 1000))
  File "C:\YTImp\python\youtrack\connection.py", line 224, in createAttachmentname)
  File "C:\YTImp\python\youtrack\connection.py", line 218, in _process_attachmnets
    raise e
urllib2.HTTPError: HTTP Error 204: No Content

:(

Dave,

Could you please clarify, are there any attachments that were successfully attached to the imported issues?
Provide us with the import log please.

Thank you.

Hi again,

Apologies for the delay - our internet access was down for a while.

Where would I find the import log?

I did some further investigation on this, and I deleted the file that had failed from the database. Another attempt at running the script succeeded more, but dropped out again with another attachment. The first attachment had been 15Mb in size but this one was only 500Kb.

If you let me know where the import log is I will send it to you.

Dave,

Import log is the output of the running script. So, you've sent us only exception last times, but could you please provide us with the full output?
Also, provide us with the youtrack.log , please (USER_HOME./youtrack/logs).

Thank you.

I'm currently running the import again to generate the import log. I'll attach here once it's done.

In the meantime, here are the two most recent youtrack logs - the "logs.1" file will include the logs from yesterday evening's attempt.

Thanks

From what I've observed, the attachments are generally being processed OK apart from a select few - I am wondering if the attachment BLOB data has been corrupted in some way in the SQL backup.
logs.7z (171KB)

Hi again, Andrey.

Please find attached the import log file as requested.

Dave
importlog.7z (6.7KB)

Another issue has come to light with this import. I'm afraid.

We have two completely independent Bugzilla databases that need to be merged into one youtrack installation. Each contains completely separate data for separate products.

The data from the first database was migrated no problem. During this process we have been discussing, data has been migrated over, but the data from the second database has been added into the original databases products.

Both databases have products with IDs 1 to 10. Issues from the second database's products 1-10 have been added into the issue lists from the first databases products 1-10, and have lost the original product information.

For example, database 1 has a product called Washing Machines with product ID 1. Issues for this are numbered 1 to 122. Database 2 has a product called Hair Dryers with product ID 1, and it has issues numbered 1 to 100.

The first database is migrated over and the Washing Machine issues are numbered 1-1 to 1-122. All is well. When the second database is migrated over, there is no product called "Hair Dryers", and the Washing Machine issues now have the Hair Dryer issues appended to the end, numbered 1-123 to 1-223.

Is there any way to stop this happening? Would it be a manual edit of the product IDs (or an SQL query to do same) of is there capacity in the migration scripts to avoid this happening?

Regards,

Dave

Hi Dave,

We've investigated your import output and youtrack.log.
Library is updated once again. We've added several cases and fix the 404 error. But some cases with 204 error are not clear from the logs. Could you please try again, I hope it becomes correctly, and even if it doesn't- we've added some extra points in import log.
So, please try again and send us the output and youtrack.log.

Thank you for your patience.

Speaking of collisions with two projects, name your project IDs with different characters. For example, Washing Machine will have ID 'WM' and Hair Dryes- 'HD'. After that, start import.

Thank you.

Thanks Andrey. I'm trying the import again and will let you know.

Regarding the collisions, it's not possible to rename the projects as you suggest. Bugzilla stores the products in a table, and the product_id is the primary key for that table. Changing the content would mean the whole issue log and structure of the database would need to be changed.

Is there another way? Could scope be added to the import script to check for an existing productID in the YouTrack database and assign the incoming product a new ID if it already exists?

Dave

Additional information: At the moment I cannot move any further on this issue as the 60-day trial license has expired. My organisation is in the process of paying for the full product, and I will action the above as soon as this process has been completed.
Apologies for the delay.