NodeJS Image Search Engine

This code requires Docker compose, which cannot be run in HF Spaces. Instead, I have briefly explained the architecture and how it works.

This repository contains the code of a NodeJS server for image search. This project was implemented to prove how to join Python code with NodeJS code using Docker. In particular, the Python code performs the search using the CLIP model and then sends the results to the NodeJS server. Then, this server displays the images that were obtained from the text search process. The specific architecture of the application can be seen in the figure below.

As it can be seen, this architecture is composed of two docker containers, one for the NodeJS server and the other one for the Python code, which is the one that performs the search process in the image dataset.

Step 1

First of all, it is necessary to extract the embeddings from the images. This is done using the text encoder from the OpenAI CLIP model. Once done, the embeddings (in this case) are stored in a CSV file (not the best way to do it, but the most effective for a small project) that should be easily replaceable with a proper database management platform. The embeddings will have a key that indicates the image path it refers to, making possible to know which images must be listed later.

Step 2

In this second step, the text query is processed using the text encoder from the same CLIP model that was previously used to obtain the image embeddings. The text embeddings are compared with all the images' embeddings, obtaining a vector distance between the text query embeddings and each image embeddings. Then, once the distances are obtained, the list of images is ordered putting the closest images first (i.e., the ones that are more likely to match with the text query).

Step 3

Once the resultant list of images is obtained, these are displayed in the browser. In this case, the code has been hardcoded to return the first 20 images with the closest distance but, of course, this could be changed and even modifiable adding a text box in the HTML template that accepts numbers.

Result

Here is how the webpage is displayed. Not the most beautiful template ever, but it does the job! Feel free to modify in the way you want and need.

Execution

You just need to clone this repository and type docker compose up in your terminal. After all the building process and so on, it should look something like this. Remember that this project not only runs a NodeJS server, but a Python server as well, in order to communicate both modules between them.