Blog

jT CarFramework CSV List Import Automation with CronJob

What is CronJob and how it will help me?

Like the Windows Task Manager, CronJob is a Linux tool that allows your server to schedule tasks to be executed at a pre-defined period of time.
We are introducing CronJob to jT CarFramework 2 Series to help you concentrate on sales and leave the inventory management to the system which will import and update your vehicles automatically.

How jT CarFramework CronJob engine works

If you are familiar with our way of doing things, you know we like to think out of the box and, at the same time, solve real life problems through technology. With that said, allow us to explain how we built this engine without using too many technical language. To make it simple for you, we will first describe what our competitors usually do.

- When your list provider (trader) uploads a CSV list to your server, the system will read this file and update the database. It would be alright just the way it is if not for the following:

  • On the front-end, the vehicle images are been loaded from a 3rd party server. If the image is removed, it will make your page unprofessional by displaying “broken” images.
  • Images are usually displayed without ALT tag. That is VERY important for SEO.
  • You are providing FREE BACK-LINKS to your list provider in ALL your inventory pages making THEM more relevant to search engines than YOU. In other words, if you are paying for SEO, your are automatically helping them too!

We could go on with the list above but, hopefully the point has been made. If you have your online inventory hosted by a 3rd party service provider you will never have the option to change things around, or even make it look better without having to pay for customization.

- Now, we will describe the process we developed to solve the above problems.

Once the list is made available, it will “normalized”. The normalization process can compress the list up to 40% of it’s original size. Think of it like ZIPPING a file.
The second step is to measure the file size. Running 4+ MB consumes a lot of server resources and, if you are using a simple shared host, it will become slow or time-out. The cronjob engine allows you to set what is the maximum “chunk” size to run. For instance, if you set to 200KB, the file will be broken in 20 parts (hence chunks) and import the first chunk to your inventory.
Each vehicle entry can hold dozens of pictures so, again, the cronjob engine allows you to set the maximum amount of images you want to save. ALL IMAGES within the list are saved to your server to avoid the problem mentioned above (SEO, broken links, etc).
Each vehicle is saved to your database and will be listed in your inventory automatically. If the vehicle already exists in your inventory, the engine will update the information accordingly.

If you set the cronjob to run twice every hour using the settings described above, you will be able to run all the chunks in 20 hours without over-using your server resources and, at the same time, hosting all the pictures.

How to setup a cron job?

CronJob can be set under LINUX SEVERS ONLY. If you are using IIS, make sure to refer to your IIS documentation.

Command lines EXAMPLES

Note: You may need to adjust /home/ to /home#/ depending on the home directory your account resides on. To view the home directory for your account simply view the stats column on the main Administration Panel page of your account and look for the home directory.

PHP

Command to run a PHP5 cron job:

php /home/username/public_html/cron.php

Optional flags are sometimes required for a PHP cron job:

php -q /home/username/public_html/cron.php

Command to use a specific php.ini file:

php -c /home/username/public_html/php.ini /home/username/public_html/myscript.php

Command to GET a remote file:

/usr/bin/GET http://www.example.com/file.php

Perl

Command to run a CGI cron job:

perl /home/username/public_html/cgi-bin/file.pl

SSH

Command to run a shell script cron job:

/bin/sh /home/username/public_html/file.sh

MySQL

Note: It is good practice to not type your password out in the follow commands but to simply use the -p flag alone and have the system prompt you for the password. This is way your password stays secure and is never on the server as plain text.

Command to import a database:

mysql -u mysql_user -ppassword database_name < backup.sql

Command to export a database:

mysqldump -u mysql_user -ppassword database_name > backup.sql

Custom Installations without Easy Apache 3 (VPS/Dedicated)

PHP

Command to run for a PHP5 cron job:

/usr/local/php5/bin/php5 /home/username/public_html/cron.php 

Command to run for a PHP4 cron job:

/usr/bin/php /home/username/public_html/cron.php

More information

Don’t hesitate in contacting us for more information through our contact form. Thank you!

3 Comments

  1. Luo Li

    Still using Linux Cron? Now you can switch to our more powerful and easier-to-use webcron FOR FREE!
    http://www.easycron.com/clickschedule

    1. Jay

      Hi Luo Li, nice to know there is a solution out there to help with the cronjob process. I’ll definitely recommend it. The only downside I see it the timeout limitation. Importing hundreds of cars on each list is very common on medium to large dealers. I would be interested to hear what you can offer. If it’s convenient for you, I can create a page about your solution on intersofts.com. (I’ll need the text content though).

      1. Luo Li

        Hi JT CarFramework,

        Thanks for your reply.

        For webcron service, I agree with you, timeout issue could be the biggest concern of our users.

        There are several ways to mitigate the impact of timeout:
        1. We suggest users who need long cron job execution time use our higher plan (our Max plan allows 1800 seconds timeout, and user can even order custom plan that has higher specification than that).
        2. If they want, they can set the script to ignore user abort (e.g. in PHP ignore_user_abort). So when the cron job timeouts in our end, the script can still continue running until it finishes.
        3. In cron job script, developer should split big cron job task into smaller ones and make it possible to perform them in separate cron jobs without messing up things.

        Thanks for the opportunity you offer of writing a post. We really appreciate it.

        Let me know if you have any suggestion or question. We can work out a best cron solution for the users

        Best Regards,

Leave a Comment

Your email address will not be published. Required fields are marked *

     

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>