Hello ALL,
i am not sure if this column is the right place for this topic.

I am setting up a searching engine to provide info searching service. The system architecture includes,

(1) Client. The user decribes his query request (some keywords) on a JSP web page.

(2) Web Server. In the back-end server, Java SERVLETs are used to receive the keywords submitted by the user. As soon as the server receives the keywords, it forward them to a PC cluster via a SOCKET link(a SOCKET client runs on the Web Server). The PC cluster takes care of executing searching operations and generating searching results (finding out relevant documents).

(3) PC cluster/DATABASE. A socket server runs on the PC cluster to receive the forwarded keywords. Once the keywords are received, PC cluster executes searching algorithm to generate searching result. Then the PC cluster returns the result to the Web Server by writing the content of the results as records (Two fields: DocumentTitle and Similarity)into a Database (DbResult). Finally, the Web Server generate a web page based on the records and present the web page to the user's client.

Since doing searching on the PC cluster takes time, the servelet on the Web Sever has to keep checking with the database DbResult to see if the result is ready SOON after it sends the keywords to the PC cluster via the SOCKET channel. If the result is ready, it begins to produce the web page (a list of Doc Title and corresponding similarity score) based on the records.

In this scheme, the servlet has to keep checking if the result is ready, consuming valuable CPU time. When there are many visitors, the web server wastes a lot of time on this meaninglesss operation.

I wonder if there is better methods/architectures to implement this system, and if there is some ways to avoid the keeping checking database operation.