The code you can download by the following link

Video unfortunately were cropped by 720p format and parts of my mac screen is not visible.
It is recommended to view it in HD mode, otherwise it will be hard to read the code on the screen.

Magento Test Driven Development Seminar – Practice Part of the Seminar from Ivan Chepurnyi on Vimeo.

Small notice:
I was entering phpunit –colors UnitTests.php for running the tests in the command line.

So what is waiting for you this Friday?

Magento Test Driven Development in Practice
Webinar on January 27

Space is limited.

Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/489066878

During 1,5 hour webinair we are going to show you how to:

• Writing Simple Tests for Magento
• Fixtures, Expectations, Data Providers
• Model/Block/Helper Tests
• Controller/Layout Tests
• Mocking and Stubbing Magento entities

And also small Questions & Answers  section for  15 mins. So be prepared for them!

Date: Friday, January 27, 2012
Time: 5:00 PM – 6:30 PM CET

After registering you will receive a confirmation email containing information about joining the Webinar.

System Requirements

• PC-based attendees
  Required: Windows® 7, Vista, XP or 2003 Server
• Macintosh®-based attendees
  Required: Mac OS® X 10.5 or newer

ps, good paint job guys!

Numbers

Before talking about features of this extension, I’d like to give some benchmark results for our extension.

Hardware used for benchmarks: AMD Athlon(tm) 64 X2 Dual Core Processor 6000+ 3GHz, 8GB DDR2

Magento instances used for calculation of performance:

• Simple: 120 Skus, 27 Categories, 3 store
• Usual: 3,912 Skus, 17 Categories, 1 store
• Large: 31,124 Skus, 197 Categories, 5 stores
• Huge: 202,628 Skus, 401 Category, 1 store

I tried to measure four full reindex processes for core rewrite indexer, but for Large and Huge instances I was too bored after waiting an hour for first run to complete, so I just killed a process. If someone wants perform such benchmarks on similar server machine, I will add results to the table.

1st run was made on clean url rewrite table, e.g. without any prepared index data.
2nd, 3rd and 4th are reindex process on existing data.

Results

As you see in the result table, difference in performance between our and core implementation is more noticeable on databases with more then 600 Skus.

Features

So does our indexer support all the features of core url rewrite one? Sure, it has such features:

• Transliteration of accented latin, German, Hebrew, Russian, Ukrainian characters to default latin characters range ([a-z]).
• Multi-store (language based) url paths for products & categories
• Nested url paths for categories & categories
• Nested url path for products that are in anchor categories and not assigned directly (Not supported by Core)
• Handling duplicated url path with adding index to it (unique-key, unique-key-1, unique-key-2, unique-key-3, etc)

System Requirements

MDBS is not supported for now.

• MySQL5.0 or higher
• Magento CE1.4.2,1.5.x,1.6.x

Installation

For now you can obtain this extension only via Git repository:
https://github.com/IvanChepurnyi/EcomDev_UrlRewrite

I will upload an extension to Magento Connect when come back from my Vacation, that will be from Oct 10th till Oct 14th.

If you have any ideas or features you would like to add to the extension, you can use Issue Tracker on GitHub.

Custom Indexer Creation Guide

Here you can see step by step implementation of a possible example with featured products indexer. As I told on seminar there is more easy and faster way to create featured products, for instance if you add “is_featured” to flat index. But this one is good for learning simple indexer creation.

You can download a full module code by clicking here.

1. Creating Product Attribute

Let’s create a product attribute, that will be used for marking our products as featured.

$this->addAttribute(Mage_Catalog_Model_Product::ENTITY, 'is_featured', array(
    'type' => 'int',
    'label' => 'Is featured',
    'input' => 'select',
    'source' => 'eav/entity_attribute_source_boolean',
    'global' => Mage_Catalog_Model_Resource_Eav_Attribute::SCOPE_STORE,
    'user_defined' => false,
    'required' => false
));

So as you see in the code, it has store view scope for making possible to define different featured products for every store view.

2. Designing Index Table

Before you implement  a new indexer you need to take care about how the index table will look like. E.g. you need to create  a table structure that will be optimized for using it on the frontend. So we are going to create a table with only two columns: store_id and product_id. Of course let’s use available Varien_Db_Ddl_Table.

// Defining table structure via object
$table = $this->getConnection()
    ->newTable($this->getTable('ecomdev_indexseminar/featured_products'))
    ->addColumn('store_id', Varien_Db_Ddl_Table::TYPE_SMALLINT, null,
                array(
                    'unsigned' => true,
                    'nullable' => false,
                    'primary' => true
                ))
    ->addColumn('product_id', Varien_Db_Ddl_Table::TYPE_INTEGER, null,
                array(
                    'unsigned' => true,
                    'nullable' => false,
                    'primary' => true
                ))
   ->addForeignKey(
        $this->getFkName('ecomdev_indexseminar/featured_products', 'store_id', 'core/store', 'store_id'),
        'store_id', $this->getTable('core/store'), 'store_id',
        Varien_Db_Ddl_Table::ACTION_CASCADE, Varien_Db_Ddl_Table::ACTION_CASCADE
    )
    ->addForeignKey(
        $this->getFkName('ecomdev_indexseminar/featured_products', 'product_id', 'catalog/product', 'entity_id'),
        'product_id', $this->getTable('catalog/product'), 'entity_id',
        Varien_Db_Ddl_Table::ACTION_CASCADE, Varien_Db_Ddl_Table::ACTION_CASCADE
    );
// Executing table creation object
$this->getConnection()->createTable($table);

Foreign keys just used for auto-cleaning the index table if product or store view was deleted. You may not add them if you need to have additional performance on index creation process.

3. Defining Indexer in Configuration

So now we going to proceed with creation indexer model itself, so let’s define it in configuration, to let Magento know about new indexer.

<config>
  <!-- other code... -->
  <global>
    <!-- other code... -->
        <index>
            <indexer>
                <featured_products>
                    <model>ecomdev_indexseminar/indexer_featured</model>
                 </featured_products>
            </indexer>
        </index>
    <!-- other code... -->
  </global>
 <!-- other code... -->
</config>

So we need just to specify our indexer code that can be used for getProcessByCode() method and model name that will realize our indexer itself. In this code snipped, indexer code is “featured_products” and model is
“ecomdev_indexseminar/indexer_featured”.

4. Creating Indexer Model

All the indexer models should be extended from Mage_Index_Model_Indexer_Abstract.

class EcomDev_IndexSeminar_Model_Indexer_Featured
    extends Mage_Index_Model_Indexer_Abstract
{
}

Then you need to define events that this indexer should process by _registerEvent method, i.e. matching entity and event types to make them available for index.

class EcomDev_IndexSeminar_Model_Indexer_Featured
    extends Mage_Index_Model_Indexer_Abstract
{
    protected $_matchedEntities = array(
        Mage_Catalog_Model_Product::ENTITY => array(
            Mage_Index_Model_Event::TYPE_SAVE,
            Mage_Index_Model_Event::TYPE_MASS_ACTION
        )
    );
}

In our case it is only save and mass product update events for catalog product.

Also lets create _construct method, that initializes our resource model (our database layer for indexer)

class EcomDev_IndexSeminar_Model_Indexer_Featured
    extends Mage_Index_Model_Indexer_Abstract
{
// .. previous lines
    protected function _construct()
    {
        $this->_init('ecomdev_indexseminar/indexer_featured');
    }
}

Then we need to add methods that will return description and our indexer name for displaying on “System -> Index Management” page. They are quite simple doesn’t require to explanation.

class EcomDev_IndexSeminar_Model_Indexer_Featured
    extends Mage_Index_Model_Indexer_Abstract
{
// .. previous lines
    public function getName()
    {
        return Mage::helper('ecomdev_indexseminar')->__('Featured Product');
    }
    public function getDescription()
    {
        return Mage::helper('ecomdev_indexseminar')->__('Indexes something');
    }
}

Then we need to realize a method that will perform a check for changes in product, that may affect rebuilding index and store that data for our indexer further process.

class EcomDev_IndexSeminar_Model_Indexer_Featured
    extends Mage_Index_Model_Indexer_Abstract
{
// .. previous lines
    protected function _registerEvent(Mage_Index_Model_Event $event)
    {
        /* @var $entity Mage_Catalog_Model_Product */
        $entity = $event->getDataObject();
        if ($entity->dataHasChangedFor('is_featured')) {
            $event->setData('product_id', $entity->getId());
        } elseif ($entity->getAttributesData()) {
            $attributeData = $entity->getAttributesData();
            if (isset($attributeData['is_featured'])) {
                $event->setData('product_ids', $entity->getProductIds());
            }
        }
    }
}

So now let take a look what it actually does:

• If it receives a product model that has “is_featured” attribute value changed, then it stores this product id for event data. Usually indexers add own prefix to values that are stored to event, to not conflict with each other.
• Then it checks if it is a massaction object, that contains a change for “is_featured” attribute as well. If our attribute changed, then store an array of product ids to reindex.
• And of course it doesn’t store anything if related data to its logic was not changed

Okey, seems our indexer registers proper data for further indexing process, so let go forward. Now it’s time to create method that runs indexer resource model method depending on entity and event types.

class EcomDev_IndexSeminar_Model_Indexer_Featured
    extends Mage_Index_Model_Indexer_Abstract
{
// .. previous lines
    protected function _processEvent(Mage_Index_Model_Event $event)
    {
        if ($event->getData('product_id') || $event->getData('product_ids')) {
            $this->callEventHandler($event);
        }
    }
}

So this method only checks if there is any data we set before, and uses callEventHandler() method that invokes proper resource model method. For instance if we have entity type “catalog_product” and event type “save” then the name of invoked method will be “catalogProductSave”, i.e. concatenated and “camelized” name from this two properties of event.

5. Creating Resource Model

So the index model is ready, the only one most important part left. It is our index model that generates all the data for index. Index module also has an abstract class for that we should extend from: Mage_Index_Model_Resource_Abstract.

class EcomDev_IndexSeminar_Model_Resource_Indexer_Featured
    extends Mage_Index_Model_Resource_Abstract
{
    protected function _construct()
    {
        // Defining the connections prefix for our resource.
        $this->_setResource('ecomdev_indexseminar');
    }
}

Also as you see it does some initialization in _construct method, for letting Magento know wich connection it may use, if there will be different connections for indexer module.

Then we are going to create the main method that will be invoked for rebuilding any kind of index, I made it flexible, because I don’t like copy-paste . The method body is quite good explained by comments in the snippet itself.

class EcomDev_IndexSeminar_Model_Resource_Indexer_Featured
    extends Mage_Index_Model_Resource_Abstract
{
// ... other code
    protected function _reindexEntity($productId = null)
    {
        // New select object for data for index retrieval
        $select = $this->_getWriteAdapter()->select();
        // Getting attribute model for retrieving its meta-information (attribute id, backend table name)
        /* @var $attribute Mage_Catalog_Model_Resource_Eav_Attribute */
        $attribute = Mage::getSingleton('eav/config')
                         ->getAttribute('catalog_product', 'is_featured');
        $select
            // Select all store views
            ->from(array('store' => $this->getTable('core/store')), array('store_id'))
            // Joining default store value
            ->join(array('default_value' => $attribute->getBackendTable()),
                         'default_value.store_id = 0', array('entity_id'))
            // Select current store value
            ->joinLeft(array('store_value' => $attribute->getBackendTable()),
                       'store_value.attribute_id = default_value.attribute_id'
                       . ' AND store_value.entity_id = default_value.entity_id'
                       . ' AND store_value.store_id = store.store_id',
                       array())
            // Limiting select by only featured products
            ->where('default_value.value = ? OR store_value.value = ?', 1)
            // Add condition if current store value is not "No" (empty or not equals to 0)
            ->where('store_value.value IS NULL OR store_value.value != ?', 0)
            // Limiting select only by our attribute data
            ->where('default_value.attribute_id = ?', $attribute->getId())
            // Do not generate values for default store
            ->where('store.store_id != ?', 0);
        // If index should be rebuild by one or some set of products
        if ($productId !== null) {
            // If single product id, then cast to array
            if (!is_array($productId)) {
                $productId = array($productId);
            }
            $select->where('default_value.entity_id IN(?)', $productId);
            // Cleaning existing rows for product to reindex
            $this->_getIndexAdapter()->delete(
                $this->getTable('ecomdev_indexseminar/featured_products'),
                array(
                    'product_id IN(?)' => $productId
                )
            );
        } else { // If it is full re-index clear all index data
            $this->_getIndexAdapter()->delete($this->getTable('ecomdev_indexseminar/featured_products'));
        }
        // Fullfil the index table from our select
        $this->insertFromSelect(
            $select,
            $this->getTable('ecomdev_indexseminar/featured_products'),
            array('store_id', 'product_id')
        );
    }
}

After we have unified method for generation of index data, we can just call it from different index event handlers:

class EcomDev_IndexSeminar_Model_Resource_Indexer_Featured
    extends Mage_Index_Model_Resource_Abstract
{
// ... other code
    // Handler for "Reindex" action in the admin panel or console
    public function reindexAll()
    {
        // Call our reindex method without parameters,
        // to rebuild all data
        $this->_reindexEntity();
    }
    // Handler for save event on particular product
    public function catalogProductSave($event)
    {
        // Passing single product id to reindex method
        $this->_reindexEntity($event->getData('product_id'));
    }
    // Handler for updating products data via massaction
    public function catalogProductMassAction($event)
    {
        // Passing an array of product ids to our reindex method
        $this->_reindexEntity($event->getData('product_ids'));
    }
}

So congratulation you have working custom indexer!

To apply this data on the fronend you can find a small example with observer for it in the code, available to download. It observes catalog_product_collection_apply_limitations_after event and calls a method on resource model that joins our index table, so limits selection results.

Also it uses collections flag functionality to add index only when it needed. So if you want to display featured products in the block, you need to set flag “featured” to a product collection, before loading the collection itself.

$collection->setFlag('featured', true);

Seminar Video

Before you are going to check this video recording from seminar, I want to excuse for some issues with the presentation, I just got cold one day before it and it was first such an online seminar when you don’t have visual contact with attendees. But anyway I hope you enjoy it

Magento Index Functionality Seminar – Sep 8 from Ivan Chepurnyi on Vimeo.

Next Seminar

Also we are planning to make it on regular basis. Our goal is to provide at least one seminar a month.
So the next seminar possible topics are presented in the voting form. The most voted topic will be prepared for the one, that will be planned for October. Most probably at the end of the month.

It is pity, but there will be only 20 places available, because of GoToTraining™ limitations. So we are starting pre-registration for this seminar with questions that will help us to select 20 attendees out of all the pre registrations. Don’t worry if you are not in selected 20 attendees, seminar will be recorded, so you can view it later.

Seminar Agenda:

• Overview of Magento Index module functionality (Workflow, Structure, Entities, etc)
• Changes of Index Module in CE 1.6
• Questions & Answers (20 minutes)

Add an event to you calendar: iCal

UPDATE 06/09/2011

We increased the maximum number of attendees on our GoToTraining account, so now everyone can join the seminar! Please hurry up, the registration closes on Sep 7th at 23:59 CEST. Here is direct registration link: http://bit.ly/op6vQ5

• Full support of Magento Enterprise features!
  Added core code support for such Enterprise Edition features:
  • Reward Points
  • Customer Balance
  • Gift Wrapping
  • Gift Receipt
  • Printed Card
• Support of Gift Messages functionality
• Support for upcoming new version of Ebizmarts SagePay extension
  Patch for current version can be issued by request.
• Removed dependency on payment template.
  Now extension uses all core templates, except a template for the order review step.

If you already bought or received extension for reviewing purposes, you can download it at “My Downloads” section on your account page in our extensions shop.

Check Our Demo Installment for seeing new version in action!

The advanced training week includes 3 different courses on the following topics:

Magento Indexers €599,-

August 2nd

• Index functionality advantages for Magento
• Index Module Overview
• Custom indexer creation
• Catalog EAV indexer (Layered Navigation)
• Price Indexers
• Product Visibility Indexers
• Product Inventory Indexer
• Catalog Search Indexer

You can find more information about the topic by viewing slides from Magento Developers Meetup in Amsterdam on June 17th, there was one hour seminar concerning it.

Behind Checkout & Sales €999,-

August 3rd

• Checkout Overview
• Quote, Quote Address & Items
• Totals
• Taxes
• Payment & Transaction
• Shipping Estimation
• Topics of the Second Day

August 4th

• Sales After Checkout
• Service Models
• Order, Order Address & Items
• Invoices
• Shipments
• Credit Memos

Really nice course for developers who want to write own payment/shipping module or write a customization to checkout process.

Third-Party Integrations €599,-

August 5th

• API vs Magento Module
• Stock Management
• Sales Export
• Importing Shipments
• Importing Creditmemos

You do not know how to write fast and lightweight export / import of data to / from third-party system? Then this course is for you!

Special Discounts!

If you are attending more than one course, then we have nice special bundled prices for you, that are valid if you register before July 25th

Magento Indexes + Behind Checkout & Sales + Third-Party Integrations €1999,-

Behind Checkout & Sales + Third-Party Integrations €1499,-

Changes in release 1.1.1

• Stored Addresses
  How many times your customers stop to fulfill order details because they forgot add something to cart? How many times they were frustrated with empty address data after coming back on checkout page? With our checkout, you will not have this problems anymore. Customer’s address details will be stored during all the checkout process, even if customer goes back to shopping cart. (Of course except secure information, like passwords, credit card details, etc)
• Smart validation
  This version comes with more smart validation that is invoked only on fields that was just changed by customer and advises him what he is entering wrong on early steps!

If you already bought or received extension for reviewing purposes, you can download it at “My Downloads” section on your account page in our extensions shop.

Check Our Demo Installment for seeing new version in action!

Magento 2.0 is not the same as Magento 1.x

As we all expected before it is not writing a new Magento version from scratch, but that doesn’t mean system will be the same in structure. They are going to perform a very deep re-factoring on it, and they already did a lot improvements in the system architecture.

Component Oriented Modules

This is one of my favorite changes that will be in new version. So what the component is for Magento 2.0? It is a set of modules that has high code coupling between each other. So you will be able to disable some components and/or replace it with your own implementation. For organizing relations between components there will be interfaces created that should be implemented in your component.

So you will be able to disable sales, checkout, customer components and can use only catalog. Or even disable catalog and use Magento as CMS… You will be able to write your own catalog implementation if you don’t need all the complexity of core one, but it will not affect other components functionality.

I really like this improvement, cant wait for seeing it in action…

Good Bye Class Alias

There will be no class alias in 2.0, all the objects will be created via createObject() factory by passing full class name. No more definitions of class aliases in configuration.

View Modularity

Someone likes it, someone not, but it is great improvement in organization of the module structure. Now your default layouts/templates/images/css/js will be in your module. There will be no base design package directory any more, all in one place app/code/[pool]/Namespace/Module/view. I suppose there will be no need to define your layout file in configuration anymore. Just create it and it works!

As for custom themes there also were added additional restrictions. All the custom themes that are created will have directories with views per module. So if you want to replace some template, layout you need to create directory with the same name as module name (Namespace_Module not Namespace/Module) and place overridden file there.

For me it seems more consistent from backend developer’s point of view, but for theme developers it will be a bit tough to change the way of customization. Anyway there will be a documentation for it, so it is not a big deal.

Infinite Themes Inheritance

One more cool feature for Magento, you will be able to create custom fallbacks for themes depending on any holiday, browser, store, etc… So you may create more clear theme structure for your customers, finally you will get rid of copy-pasted data.

Can not wait for it?  It is possible in current Magento version as well, you just need to perform a small customization to core/design_package model, override _fallback method and call parent with additional $fallbackScheme rules.

ORM

They are planning to support more than one database engine so all the resource model level logic will be re-factored. It will be  a separate service level, so maybe in future they will just switch to Doctrine ORM… Maybe in Magento 3.0

Good Bye EAV?

It is not their final decision, but they have plans to replace EAV with more simple key/value implementation for RDBMS and you will have an option to select NoSQL adapter instead of RDBMS. I really appreciate this change, if they finally decide to make it in such a way, because EAV is just adding complexity and performance issues.

Integration Tests

They are writing integration test (non-isolated) that covers legacy functionality in core. Currently they have about 47% code coverage for Mage_Catalog  and 53% for Mage_Core.

That’s great news, because it is important to have some proof that the core code will work after re-factoring. Of course it would be better to use TDD practices in core, but I suppose they can implement them only after covering existing functionality.

Different Directory Structure

Now there will be only 5 directories in Magento root instead of current 9. All the publicity available files will be placed in “pub” directory, instead of current “media”, “js”, “skin”, “error”. So you will be able easily place your Magento application files outside of webserver document root. In “pub” will be placed “jslib” (instead of “js” in root), “error” (error reporting), “media” (uploaded media files), “skins” (theme related public files).

Magento Contributions

Now there will be more clear process of involving community into developing Magento 2.0. We will be able to see all the process in their public Jira and develop for core if we want.
For instance I signed MCA agreement quite a long time ago and would like to take a part in developing 2.0 features. Hope their Jira and SVN repo will be available soon.

You can also refer to Dmitriy’s slides for something that I haven’t mentioned here (some text in Russian, but mostly English):
http://www.meet-magento.by/fileadmin/user_upload/meet-magento.belarus/Magento_2_By_SorokaDmitry.pdf