Will sending an HTTP Header with Accept: text/html only download text from the page? -
I'm writing a simple crawler, and ideally to save bandwidth, I only download text and links Want panna Can I use HTTP Headers? I'm confused about how they work. You are on the right track to solve the problem. I'm not sure how much you already know about HTTP Headers, but basically just a string formatting for an HTTP header web server - it follows a protocol - and that Very simple in the aspect You write a request, and receive a response. The requests you see in the Firefox plugin look like things seen in the LiveHTTPHeaders. I wrote a small post on my site that shows you how you can write a request for a web server and read the answer later. If you accept only text / html, you will only accept the subset made available on the web (yes, it will "optimize" to some extent your script). Note that this example is actually low level, and if you are writing a spider, then you can use an existing library such as curl or other tools that provide the langua...