• 9
name

A PHP Error was encountered

Severity: Notice

Message: Undefined index: userid

Filename: views/question.php

Line Number: 191

Backtrace:

File: /home/prodcxja/public_html/questions/application/views/question.php
Line: 191
Function: _error_handler

File: /home/prodcxja/public_html/questions/application/controllers/Questions.php
Line: 433
Function: view

File: /home/prodcxja/public_html/questions/index.php
Line: 315
Function: require_once

This is the script:

import requests
import json
import urlparse
from requests.adapters import HTTPAdapter

s = requests.Session()
s.mount('http://', HTTPAdapter(max_retries=1))

with open('proxies.txt') as proxies:
    for line in proxies:
        proxy=json.loads(line)

    with open('urls.txt') as urls:
        for line in urls:

            url=line.rstrip()
            data=requests.get(url, proxies=proxy)
            data1=data.content
            print data1
            print {'http': line}

as you can see, its trying to access a list of urls through a list of proxies. Here is the urls.txt file:

http://api.exip.org/?call=ip

here is the proxies.txt file:

{"http":"http://107.17.92.18:8080"}

I got this proxy at www.hidemyass.com. Could it be a bad proxy? I have tried several and this is the result. Note: if you are trying to replicate this, you may have to update the proxy to a recent one at hidemyass.com. They seem to stop working eventually. here is the full error and traceback:

Traceback (most recent call last):
  File "test.py", line 17, in <module>
    data=requests.get(url, proxies=proxy)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 454, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 144, in resolve_redirects
    allow_redirects=False,
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host=u'219.231.143.96', port=18186): Max retries exceeded with url: http://www.google.com/ (Caused by <class 'httplib.BadStatusLine'>: '')

Looking at stack trace you've provided your error is caused by httplib.BadStatusLine exception, which, according to docs, is:

Raised if a server responds with a HTTP status code that we don’t understand.

In other words something that is returned (if returned at all) by proxy server cannot be parsed by httplib that does actual request.

From my experience with (writing) http proxies I can say that some implementations may not follow specs too strictly (rfc specs on http aren't easy reading actually) or use hacks to fix old browsers that have flaws in their implementation.

So, answering this:

Could it be a bad proxy?

... I'd say - that this is possible. The only real way to be sure is to see what is returned by proxy server.

Try to debug it with debugger or grab packet sniffer (something like Wireshark or Network Monitor) to analyze what happens in the network. Having info about what exactly is returned by proxy server should give you a key to solve this issue.

  • 34
Reply Report
    • Doesn't "max retries exceeded with url" provide any possible clue? What could that mean? It sounds like the proxy tried to give a request to the web server but it kept telling it to try again (cause of some error).
    • @BigBoy1337 not really. Your trackback can be interpreted as: failed to perform request ("Max retries exceeded with url [...]" part of the error message) as a result of httplib.BadStatusLine exception raised somewhere before ("Caused by [...]" part of the error message). Note that there is no info about the reason why httplib.BadStatusLine was raised (except documentation).
      • 2
    • I am not using httplib, unless it is included in the request library? Aside from this, are you saying that the my request goes through the proxy server to the webpage, back to the proxy server, and then what that proxy server tries to relay to me is unreadable?
    • @BigBoy1337 about httplib: seems like it is used indirectly (in any case you do have exception from it in your traceback). About "are you saying that the my request goes through the proxy server to the webpage, back to the proxy server, and then what that proxy server tries to relay to me is unreadable?": it is possible but not necessary. All we know at the moment is that reply from proxy is not valid. ...
      • 1
    • ... It is possible that proxy encountered some internal error even before delivering request to final web server and thus proxy replied with invalid reply. It is also possible that this internal error happened in proxy after web server replied with valid reply to the proxy. And on top of that it is possible that web server replied to the proxy with invalid rely which in turn resulted in invalid reply from proxy. As I've said in my answer - the most straightforward way to figure out what is the root cause is to fetch more data about what was actually replied from proxy.

Maybe you are overloading the proxy server by sending too much requests in a short period of time, you say that you got the proxy from a popular free proxy website which means that you're not the only one using that server and it's often under heavy load.

If you add some delay between your requests like this :

from time import sleep

[...]

data=requests.get(url, proxies=proxy)
data1=data.content
print data1
print {'http': line}
sleep(1)

(note the sleep(1) which pauses the execution of the code for one second)

Does it work ?

  • 8
Reply Report
    • Request always adds the original reason for teh exception at the end. requests.exceptions.ConnectionError: HTTPConnectionPool(host=u'219.231.143.96', port=18186): Max retries exceeded with url: google.com (Caused by : '') In this example. (Caused by : '') That means it wouldn't be a problem with Timing more unhandled HTTP as par the answer provided.
      • 2
    • @MattPsyK I've had this "BadStatusLine" exception many times with a popular website (running standard Apache) when I sent too much requests at the same time (the sleep() trick worked for me), so maybe it's the same issue here...

Trending Tags