I’ve been working on a freelance project to sync products and collections data betwen an Airtable and multiple Shopify stores.

Project overview

I had a client where they have multiple Shopify stores and multiple product and collection linked to each Shopify store and other e-commerce platforms.

The client already a script written in Python by another developer but the client wasn’t happy with it. He reachout to me to rebuild the system to make it more reliable and scalable.

I had to build a script to generate a product image for each store and product variant based on an image base. (it was an apparel store and each store has their own brand but share similair products).

Also, I need to build a script to sync data between Shopify and Airtable (Product upload, update and delete with their varaints and metafields , Collection upload, update and delete).

Challenges

First challenge was what infrastructure will be using to build this project. I used digital occean droplets but this approach wasn’t scalable and cost efficent for client needs.

Second challenge, I didn’t had much knowledge about AWS services and from a consulatation with an AWS expert. We managed to find an infrastucte that will fits the client needs.

Third challenge was how to we can make the scripts run faster because the client will have multiple stores.

Solution

First things I did, I build an Airtable base to store all Shopify stores data (product base, API key and other data points). Also, I added a button to tigger an action like image generation, product upload etc...

When the client trigger the button. I send a new queue message (AWS SQS) with type of action and other data. For example, if we trigger an action for image generation it’ll trigger a lambda function to generate images.

I build an image generator script using Node JS and jimp for image processing and pulling image data from product image airtable base where we have all data like product image, logo, positions and colors. (AWS Lambda function)

After that, I need to upload images to S3 to access later on using Shopify upload function and linked to a Shopify product and variants images. (AWS S3).

I used AWS EC2 built on top of AWS ASG (Auto scalling group) for long polling for SQS queue message (It does scalle based on number of messages). When a new Shopify product upload has been catched. we run a script to generate products data from store’s product base and send data to Shopify and run Shopify bulk operation mutation.

We’ll be waiting for webhook call from Shopify that product upload has been done. Using Node js server, we’ll sync data from Shopify response to store’s product base like product ID and API errors.

Conclusion

This project was a challenging one for me. I learned so much things while buidling it like AWS services and Shopify bulk opeartions.

• Shopify API