multithreading - Java ThreadPool usage -


I am trying to write a multithreaded web crawler.

My main entry class has the following code: 

  Executable service exec = Executor. NewfixedThreadPool (numberOfCrawlers); Whereas (true) {url = frontier.get (); If (URL == empty) return; Exec.execute (new URLCrawler (this, URL)); }

The URL crawler brings the specified URL, removes the HTML and sends the link back to the margin.

A threshold is the line without any URL. The problem is how to write the get () method. If the queue is empty, then any URL crawlers should wait until the end and then try again the queue is empty and there is currently no active url crawler when it should be blank.

My first idea was to use an atomenter to calculate the current number of working crawlers and to calculate an auxiliary object to inform. () At the beginning of the call, each crawler increases the number of current working URL crawlers, and decreases when it comes out, and inform the object that it has completed.

But I have read that (inform) / notify () and wait () thread to some extent to exclude methods.

What should I use in this work pattern? This is similar to M producer and N consumers, the question is how to handle the evolution of producers.

I think the use / notification is appropriate in this case. Any straight j.u.c. To use it, it can not think any way forward. In a class, let's call the coordinator: 

  private final integer number of crawlers; Private Ent Wait; Public Boolean Triangle () {synchronize (this) {wait ++; If (Waiting & gt; = numOfCrawlers) {// Everyone is waiting, return falsehood; } and wait); // fake wake-up is ok / waited for any reason again - wait; Back true; }} Public is Zero () (synchronized) (this) {notifyAll (); }}

Then, 

  Executable service exec = Executor. NewfixedThreadPool (numberOfCrawlers); Whereas (true) {url = frontier.get (); If (url == faucet) {if (Coordinator.shouldTryAgain ()) // // All threads are waiting; No possibility of new jobs return; } Else {// It is possible that other jobs are retry; }} Exec.execute (new URLCrawler (this, url)); } // while (true)

-

Comments

Post a Comment

Popular posts from this blog

Eclipse CDT variable colors in editor -

I hope this is a good question to ask here. This programming is related, so I thought it would be better than the superuser, so I am using the CDT C ++ Eclipse plugin. I know how the editor changes colors for some things But is there a way to do that all the variables are different colors? For example in KDEValue, it sets the local variables and square variables in different colors and bold square squares. Can I copy it to CDT? Eclips provides some level of color customization for text editor syntax You can see it on the following: There are many elements, including the Window> Preferences> C / C ++> Editor> Syntax color code that can be changed, assembly, Comments, Preprocessor and Doxian Under the code, it is possible to change the color of some variable types, such as global variables, local variable announcements, local variable references, and parameter variables. But I do not think the code is capable of changing the color of each variable declared....
Read more

AJAX doesn't send POST query -

$ (दस्तावेज़) .ready (function () {function ajaxselectrss (rssurlvar) {var ajaxRequest; // वेरिएबल जो अजाक्स को संभव बनाता है! कोशिश करें कि {// ओपेरा 8.0+, फ़ायरफ़ॉक्स, सफारी अजाक्स अनुरोध} = नया एक्सएमएलएचटीपीआरइवेस्ट ();} कैच (ई) {// इंटरनेट एक्सप्लोरर ब्राउज़र्स {ajaxRequest = new ActiveXObject ("Msxml2.XMLHTTP");} पकड़ो (ई) {try {ajaxRequest = new ActiveXObject ("Microsoft.XMLHTTP");} कैच (ई) {// कुछ गलत चेतावनी ("आपका ब्राउज़र तोड़ दिया!"); वापसी झूठी;}}} // एक बनाएं फ़ंक्शन जो सर्वर से भेजे गए डेटा प्राप्त करेंगे ajaxRequest.onreadystatechange = function () {if (ajaxRequest.readyState == 4) {var ajaxDisplay = document.getElementById ('news'); ajaxDisplay.innerHTML = ajaxRequest.responseText;}} / / Var rssurlvar = $ (this) .attr ("title"); var queryString = "rurl =" + rssurlvar; var urltofile = "rssget.php"; ajaxRequest.open ("पोस्ट", urltofile, सच); ajaxRequ Est.setRequestHeader ("सामग्री-प्र...
Read more

wpf - Custom Message Box Advice -

Well, I'm using a custom window box that comes with some custom control boxes that accompany the text The producers who are displayed are called. I have a defined event, which has been subscribed through the original class, it gets burnt when clicking on the button. However, I can not see how to use it effectively, I would like to return one yes, whether yes or not, but obviously my code will continue to execute, so this Below is some example code to clarify the problem that the method has subscibed by clicking on the button. Message box window Public partial class custom message box: window {public representative message messagebox object sender, event events e); Public event messagebox handler messagebox document; Public Customs Box () {Initial group (); } Public Customsbox (String Message) {InitializeComponent (); This.txtdescription.Text = Message; } Public Customsbox (String Message, String Title, String First BTNtext) {Initialization (); This.lbltitle.Content =...
Read more