next »

[BobRay]

I've been looking at porting an existing site to Revolution to give it a shakedown. The site has manager users, web users, MaxiGallery, Wayfinder, file uploads, custom snippets, Personalize, Ditto, SPForm, and multiple templates so it would give Revolution a good ride. I have the latest Revolution up and running but with no content.

Obviously, I can cut and paste everything into the new site, but I'm wondering if I can use some shortcuts (and possibly develop a conversion script).

The site_content table looks to be the same except for the addition of the class_key and context_key fields and the renaming and restructuring of the contentType/content_type field. There are also a few changes from int(1) to tinyint(1) and vice versa although I'm guessing those shouldn't be an issue.

Is it possible to dump the site content table and import it into the new site after editing the dump to drop contentType, add the three new fields, and modify the snippet and chunk calls appropriately? Or would that be a suicidal undertaking? If it's possible, what should go in the new fields?

I imagine there would have to be some editing of the permissions numbers and the groups would have to be done from scratch.

Is the site_content table modular enough that it can be imported on its own, or does document creation modify other tables that would make this move unwise?

Are there any other tables that could be imported this way (member_groups, membergroup_names, document_groups, documentgroup_names, site_templates, site_snippets, site_htmlsnippets?

Any thoughts appreciated.

Bob

[OpenGeek]

I've been looking at porting an existing site to 0.9.7 to give it a shakedown. The site has manager users, web users, MaxiGallery, Wayfinder, file uploads, custom snippets, Personalize, Ditto, SPForm, and multiple templates so it would give 0.9.7 a good ride. I have the latest 0.9.7 up and running but with no content.

Obviously, I can cut and paste everything into the new site, but I'm wondering if I can use some shortcuts (and possibly develop a conversion script).

First, checkout the class modTranslator (a wrapper for the utility class modTranslate095), and an example conversion script I used to jump-start the process of translating Wayfinder 2.0 to Revolution.  After running this, only a very few things had to be adjusted in the script, and these are things that will either be fully documented in a migration guide or refactored/fixed so the change is not necessary when we finally release this.

The site_content table looks to be the same except for the addition of the class_key and context_key fields and the renaming and restructuring of the contentType/content_type field. There are also a few changes from int(1) to tinyint(1) and vice versa although I'm guessing those shouldn't be an issue.

Is it possible to dump the site content table and import it into the new site after editing the dump to drop contentType, add the three new fields, and modify the snippet and chunk calls appropriately? Or would that be a suicidal undertaking? If it's possible, what should go in the new fields?

Is the site_content table modular enough that it can be imported on its own, or does document creation modify other tables that would make this move unwise?

The aforementioned modTranslator will handle conversion of old tags to new accurately in probably 95% of all cases, with most of the false positive matches occurring in snippets where valid PHP code can match some of the legacy tags. 

As for the site content, just dump using complete and extended insert options, import, and add a class_key of modDocument (or modWebLink) and match the content_type fk to the appropriate match in the content_type table, which will likely be 1 (which should be the HTML content type record) for most modResources (modDocument, modWebLink, etc.); you can actually use contentType to find a match in content_type table as the value in site_content.contentType will match the content_type.mime_type field (these are kept in sync for now to allow new functionality without sacrificing legacy dependency on the contentType column).

I imagine there would have to be some editing of the permissions numbers and the groups would have to be done from scratch.

This is a whole other can-of-worms, and I'm working on migration strategies to ease the transition along with scripts to help automate these strategies when moving to the new security framework I've just recently implemented on top of the merged web/manager user model.

Are there any other tables that could be imported this way (member_groups, membergroup_names, document_groups, documentgroup_names, site_templates, site_snippets, site_htmlsnippets?

Start with the translator class and your manual site_content (i.e. modResource) import and we'll go from there.

The installer will eventually include interfaces to these and possibly other tools to help most 0.9.6'ers easily migrate their sites through a series of steps born out of best practices we develop here in collaboration.

[BobRay]

Excellent!

Just a couple of quick questions to start out:

1. I can't find the actual translate() function. Did I miss it, or is it in another file?

2. Is the script meant to work on the SQL dump file for the whole site, selected tables, or something else?

Bob

[OpenGeek]

For some reason I missed the topic notification until now; sorry BobRay...

1. I can't find the actual translate() function. Did I miss it, or is it in another file?

translateSite() is the actual function that performs the translation.  You can have it just create a log of the changes it would make, or run it to actually modify the database and/or file content you specify.

2. Is the script meant to work on the SQL dump file for the whole site, selected tables, or something else?

This API allows you to include any tables/fields represented by/in xPDOObject classes, directories (with optional file extension filters), and/or specific files.

I'm thinking separate migration packages for coming from previous releases (oooh, and/or other products maybe?) might be the way to go, rather than trying to stuff all that into the regular installer.

[BobRay]

For some reason I missed the topic notification until now; sorry BobRay...

Topic notification has been a little quirky for me too lately

1. I can't find the actual translate() function. Did I miss it, or is it in another file?

translateSite() is the actual function that performs the translation.  You can have it just create a log of the changes it would make, or run it to actually modify the database and/or file content you specify.

This is what I was looking for (and still haven't found):

Code:

$parser->translate($content)

It appears to be where the real work is done and where most new code/bug fixes would go.

It comes from here:

Code:

function getParser() {
        if (!is_object($this->parser) || !is_a($this->parser, 'modParser095')) {
            $this->parser= & $this->modx->getService('parser095', 'modParser095');
        }
        return $this->parser;
    }

but I don't know if parser095 and modParser095 are built-in or your own code (which I'd need for the next step).
I'm feeling particularly dense this morning so I hope these aren't stupid questions.

It's just that I don't see any code in what you've shown me that actually does the translation of, say,

Code:

{{chunk}} to [[$chunk]] or [!snippet!] to [[!snippet]].

I'm thinking separate migration packages for coming from previous releases (oooh, and/or other products maybe?) might be the way to go, rather than trying to stuff all that into the regular installer.

Agreed.


 Bob

[OpenGeek]

modParser095 is what you are looking for BobRay, as included by the modTranslate095 code you alluded to...

[BobRay]

modParser095 is what you are looking for BobRay, as included by the modTranslate095 code you alluded to...

Thanks, it's starting to make sense to me now.

Bob

[BobRay]

Jason,

Do you think a simple GUI front-end for this would be worth the effort?

Bob

[rthrash]

BobRay, definitely!

[Jesse R.]

BobRay,

If you could also try and document your efforts and either make a posting here or a Wiki posting.  I would be very interested in your efforts or a tutorial

[splittingred]

Do you think a simple GUI front-end for this would be worth the effort?

Excellent idea!

Also, you might be interested in how to do this in the 097 manager with Ext2. We're putting up a documentation/tutorial site for this:

http://docs.modxcms.com/modext/

It's in-progress, but should help a lot with how to use modExt (MODx's Ext implementation) properly and effectively.


Also, you'll probably want to look into modAction and modMenu for creating the controllers and menu items for the section.

[OpenGeek]

Are we thinking an inside the manager interface is the right way to present this?  i.e. install the new MODx and then run the migration script on an old database/file-system location?

[rthrash]

Jason I think that sounds like a great idea in getting the site up and "proving" the core functions. Then the old site could be "imported". Would be pretty darned cool and also facilitate sucking in those other types of sites as you mentioned here/elsewhere ... slurp up your Wordpress blog entries under a specific container anyone? Ready to migrate from Joomla/Nuke/whatever? Fill out your DB credentials in this form, or locate your SQL dump and press this button ...

[BobRay]

Are we thinking an inside the manager interface is the right way to present this?  i.e. install the new MODx and then run the migration script on an old database/file-system location?

I was thinking standalone app., since the user will have to go outside of MODx to dump the SQL from the old site.  It could certainly be done in the Manager but that would slow development some and make debugging more difficult.

If it works as a standalone, could it be integrated into the Manager later?

Bob

[OpenGeek]

I was thinking standalone app., since the user will have to go outside of MODx to dump the SQL from the old site.  It could certainly be done in the Manager but that would slow development some and make debugging more difficult.

There is no SQL dump involved.  Everything in this script is done via xPDO objects or directly to files.  And it could be distributed separately, but would be useless without the core install available and running and the app configured to point at the core location.

[BobRay]

I was thinking standalone app., since the user will have to go outside of MODx to dump the SQL from the old site.  It could certainly be done in the Manager but that would slow development some and make debugging more difficult.

There is no SQL dump involved.  Everything in this script is done via xPDO objects or directly to files.  And it could be distributed separately, but would be useless without the core install available and running and the app configured to point at the core location.

I was referring to the dump of the old site to be migrated. The script isn't intended to go and extract the old site content is it?  And, as a user, I'd be more comfortable having the program work on the SQL dump files and then taking a look at them before importing them into the 0.9.7 site.

I didn't see anything in the code that looked like it required the core install to be *running* for it to work, just available. It looked to me like the script created its own instance of $modx in convert.php:

Code:

require_once (MODX_CORE_PATH . 'model/modx/modx.class.php');
$modx= new modX();

Anyway, I'll start looking at developing it inside the Manager if that's the prevailing thinking.  I was hoping I could put off learning about the ext stuff a little longer 

Bob

[OpenGeek]

Ok, let's step back.  There are a large number of ways for this to be dealt with, but here is the vision I had...

• You install and configure 0.9.7
• You grab the 0.9.6 migration pack from a manual download site, or via the Workspace manager (remote download/install)
• The migration pack includes the relevant interfaces for the manager or a standalone web interface (maybe a context?), along with an xPDO model representing the 0.9.6 tables (or a process to generate the model for them using the xPDOGenerator API's)
• You access the interface, select the classes that represent the tables you want to convert (probably with a recommended set of defaults), along with any directories/files you want to convert.
• It provides an interface to specify and review dry-run's recorded to the log
• It provides the actual command to perform the translation and does it.

The question becomes, do we use the translator as is and have it update the old tables, then perform an import, or do we combine these two steps and have the translator apply to the table content as we move it to 0.9.7 structures.

[BobRay]

Ok, let's step back.  There are a large number of ways for this to be dealt with, but here is the vision I had...

Good idea (stepping back, I mean).

[1] You install and configure 0.9.7
[2] You grab the 0.9.6 migration pack from a manual download site, or via the Workspace manager (remote download/install)
[3] The migration pack includes the relevant interfaces for the manager or a standalone web interface (maybe a context?), along with an xPDO model representing the 0.9.6 tables (or a process to generate the model for them using the xPDOGenerator API's)
[4] You access the interface, select the classes that represent the tables you want to convert (probably with a recommended set of defaults), along with any directories/files you want to convert.
[5] It provides an interface to specify and review dry-run's recorded to the log
[6] It provides the actual command to perform the translation and does it.

[7]do we use the translator as is and have it update the old tables, then perform an import, or do we combine these two steps and have the translator apply to the table content as we move it to 0.9.7 structures?

1 - no question about this
2 - or the various migration packs are install options (maybe later)?
3 - ok
4 - I've got some confusion here, but some ideas about it below.
5 - ok
6 - ok
7 - I would think the former for starters. We could always add the option to automate the import later.

Here's what I'm thinking, based on your vision.

The user:

1. Gets the migration pack (manually as a first step, then, in a future version, in the Workspace Manager, and/or during installation)..
2. Gets both directory structures (0.9.6 and 0.9.7) on the same machine in different subdirectories.
3. Does an SQL dump/export of the old site and puts it in the root of the 0.9.6 MODx directory referred to in step 2.
4. Runs the migration script and selects the tables in the DB to be translated or moved (with sensible defaults and some things off limits).
5. Clicks to execute the script that copies all the selected tables in the big SQL dump file to a new .sql file, translating as it goes.
6. Gets the option to view (and edit?) the new .sql file (that's the dry run for this step).
7. Imports the new .sql file into the 0.9.7 DB. (or we do it for the user?).
8. Specifies the two root directories and then specifies the parts of the 0.9.6 directory structure to be copied/translated/etc.
9. Gets the option to see a window with a dry run of the directory/file conversion/copy.
10. Ok's the execution of the directory/file conversion/copy.
11. Gets down on his or her knees and thanks us for the opportunity to use the migration pack. 

I'm seeing steps 4-7 on one screen and then 8-10 on another to keep the user from being overwhelmed by the complexity.

Step 8 is the monster, IMO, especially since we don't know what directories the user has, and it will be hard to know ahead of time which directories should be merged, overwritten, deleted, or created. Also, which directories contain files that need translation (can we just offer to translate any files that end in .php?).

Maybe a big directory tree with check boxes next to each item (copy, merge, ignore) and a separate "translate .php files" box? Do we also need a "copy to location" choice?
Maybe two directory trees, one with stuff we know for sure what to do with and one with everything else (or the ones we know what to do with are grayed out in the one big tree?)

Or is this *way* too ambitious? Rahul says MODx can do air traffic control for the space shuttle. Is this the moral equivalent? 

Bob

[rthrash]

Anyway, I'll start looking at developing it inside the Manager if that's the prevailing thinking.  I was hoping I could put off learning about the ext stuff a little longer 

Bob

Hope some work Shaun's been doing helps there: http://docs.modxcms.com/modext/

The ideas being discussed above sound really great. For large SQL files that might time out on a busy server, we might think about breaking it into chunks and reloading the page between chunks. SMF handles its DB maintenance operations this way and I've seen some operations take upwards of 5 minutes to complete but never have seen them timeout (due to the reloads and spawning new operations for lack of a better way to phrase it).

[OpenGeek]

Alright, 50 lashes to the next person that mentions SQL files.  Forget SQL files, they will not be involved here at all.  And yes, this would need to be broken up into smaller, practical pieces if run through a web UI.