TODO

- some servers have a list of IP addrs, not just one
  If one of those did not connect (check ECONNREFUSED), try the other ones.

- empty page on search at http://gdataonline.com/seekhash.php

- functional tests

- rework the HTML GUI using styled checkboxes.

- wc.proxy <client:receive  persistent server http://download.daemlich.net/no_hotlinkers33/volkswagen-getaway.mov> (32, 'Broken pipe')
Traceback (most recent call last):
  File "/home/calvin/projects/webcleaner/wc/proxy/Connection.py", line 126, in handle_write
    num_sent = self.send(data)
  File "/home/calvin/projects/webcleaner/wc/proxy/Dispatcher.py", line 350, in send
    raise                    
  File "/home/calvin/projects/webcleaner/wc/proxy/Dispatcher.py", line 342, in send
    result = self.socket.send(data)
error: (32, 'Broken pipe')   
Traceback (most recent call last):
  File "webcleaner", line 196, in ?
    main(sys.argv[1:])       
  File "webcleaner", line 192, in main
    return main_posix(configdir, filelogs, args)
  File "webcleaner", line 159, in main_posix
    wc.start.wstartfunc(confdir=configdir, filelogs=filelogs)
  File "/home/calvin/projects/webcleaner/wc/start.py", line 59, in wstartfunc
    wc.proxy.mainloop(handle=handle, abort=abort)
  File "/home/calvin/projects/webcleaner/wc/proxy/__init__.py", line 168, in mainloop
    proxy_poll(timeout=max(0, run_timers()))
  File "/home/calvin/projects/webcleaner/wc/proxy/__init__.py", line 122, in proxy_poll
    x.handle_read_event()    
  File "/home/calvin/projects/webcleaner/wc/proxy/Dispatcher.py", line 422, in handle_read_event                           
    self.handle_read()       
  File "/home/calvin/projects/webcleaner/wc/proxy/Connection.py", line 95, in handle_read
    self.process_read()      
  File "/home/calvin/projects/webcleaner/wc/proxy/HttpServer.py", line 214, in process_read
    if self.delegate_read(): 
  File "/home/calvin/projects/webcleaner/wc/proxy/StatefulConnection.py", line 34, in delegate_read                        
    getattr(self, 'process_'+self.state)()
  File "/home/calvin/projects/webcleaner/wc/proxy/HttpServer.py", line 467, in process_content
    self.client.server_content(data)
  File "/home/calvin/projects/webcleaner/wc/proxy/HttpClient.py", line 516, in server_content
    assert self.server, "%s server_content(%r) had no server" % \
AssertionError: <client:closed   being read> server_content('\xfc\

- Mac OS X support for network interface list:
  http://lists.helixcommunity.org/pipermail/client-dev/2004-May/001032.html

- SSL gateway is broken.
  Use case:
  https://bugzilla.mozilla.org/show_bug.cgi?id=217527

- invalid js url:
http://www.burstnet.com/cgi-bin/ads/ad8328a.cgi/v=2.0S/sz=468x60A|728x90A/65740/RETURN-CODE/JS/
http://adstome.com/phpads/adjs.php?n=864959011&what=zone:5&exclude=,

- squid uses an external auth configuration, we could use that too

- http://www.sane.com/demo/NetTracker/intranet/index.html
  JS navi not there

- ssl certificate verification

- memory leaks while operating the proxy

- allow filtering/blocking by MIME type

- bfilter has Flash parse and generation library, I want that too .)

- test NTLM authentication
  Mozilla >= 1.6b can test NTLM server auth, squid can test client auth

- NTLM: restore old method type when method was not GET

- NTLM: reconnect to parent proxy after first 407 error (squid closes
  the connection)

- unicode parsing for flex
  the HTML parser (written with flex) does not support unicode, so
  sometimes you'll see '?' chars in the HTML output.

- IPv6 support (there are some stubs already, but untested, and DNS
  library is missing IPv6 support)
  Big problem: how do we decide when to send out IPv6 DNS queries?

- use IPpy.py

- Problem on WinXP box:
 The instance's SvcRun() method failed 
   File "C:\Python24\Lib\site-packages\win32\lib\win32serviceutil.py",
 line 715, in SvcRun
     self.SvcDoRun()
   File "C:\Python24\Lib\site-packages\wc\win32start.py", line 74, in
 SvcDoRun
     filelogs=self.filelogs)
   File "C:\Python24\Lib\site-packages\wc\start.py", line 59, in
 wstartfunc
     wc.proxy.mainloop(handle=handle, abort=abort)
   File "C:\Python24\Lib\site-packages\wc\proxy\__init__.py", line 168,
 in mainloop
     proxy_poll(timeout=max(0, run_timers()))
   File "C:\Python24\Lib\site-packages\wc\proxy\__init__.py", line 122,
 in proxy_poll
     x.handle_read_event()
   File "C:\Python24\Lib\site-packages\wc\proxy\Dispatcher.py", line 424,
 in handle_read_event
     self.handle_read()
   File "C:\Python24\Lib\site-packages\wc\proxy\Connection.py", line 95,
 in handle_read
     self.process_read()
   File "C:\Python24\Lib\site-packages\wc\proxy\dns_lookups.py", line
 567, in process_read
     callback(self.hostname, DnsResponse('found', ip_addrs)) 
 exceptions.TypeError: 'NoneType' object is not callable

- > post-install script gives this error and than after I can't get ssl to work:
Traceback (most recent call last):

 File "C:\Python24\Scripts\webcleaner-certificates", line 113, in ?

   main(sys.argv[1:])

 File "C:\Python24\Scripts\webcleaner-certificates", line 105, in main

   install_ssl_certs(configdir)

 File "C:\Python24\Scripts\webcleaner-certificates", line 61, in 
install_ssl_certs

   wc.proxy.ssl.create_certificates(configdir)

 File "C:\Python24\Lib\site-packages\wc\proxy\ssl.py", line 109, in 
create_certificates

   careq = createCertRequest(cakey, CN='Certificate Authority')

 File "C:\Python24\Lib\site-packages\wc\proxy\ssl.py", line 179, in 
createCertRequest

   setattr(subj, key, value)

OpenSSL.crypto.Error: [('asn1 encoding routines', 'ASN1_mbstring_ncopy', 
'invalid universalstring length')]

- Streaming clamav scanning, try to detect virii immediately
  Note: clamav does _not_ support streaming, so this is currently not
  possible. We have to wait until all data is received, then scan it,
  and only after that we can be sure the data is virus-clean.

- Remove all comments from JavaScript: use spidermonkey scanner, and
  reconstruct the script with token data.
