Make everything Unicode

After #2514 HttpResponse breaks with headers set from DB. With the following view code

def profile_dl_pgp_key(request, user_id):
    profile = get_object_or_404(UserProfile, user__pk=user_id)
    
    response = HttpResponse(mimetype='text/plain')
    response['Content-Disposition'] = 'attachment; filename=%s_public_pgp_key.asc' % profile.user.username
    response.write(profile.public_pgp_key)

    return response

this error occurs

Mod_python error: "PythonHandler django.core.handlers.modpython"

Traceback (most recent call last):

  File "/usr/local/lib/python2.4/site-packages/mod_python/apache.py", line 299, in HandlerDispatch
    result = object(req)

  File "/usr/local/lib/python2.4/site-packages/django/core/handlers/modpython.py", line 163, in handler
    return ModPythonHandler()(req)

  File "/usr/local/lib/python2.4/site-packages/django/core/handlers/modpython.py", line 146, in __call__
    populate_apache_request(response, req)

  File "/usr/local/lib/python2.4/site-packages/django/core/handlers/modpython.py", line 154, in populate_apache_request
    mod_python_req.headers_out[key] = value

TypeError: table values must be strings

Key in mod_python_req.headers_out[key] is a unicode string we got from psycopg2, so it breaks.

As a fix patch below could be used.

#2716 was closed in favor of this ticket.

+1 for Django unicodization

small fix for capfirst filter to enable unicode strings

#2376 (only ascii support in initial sql) also closed in favour of this ticket.

In #393, filters are fixed to work with unicode strings (but currently converts objects not type basestring with str() still for now).

duplicates: #3398 (send_mass_mail), #3387 (db API for filtering queries)

in #3398 the problem was that I didn't use unicode anywhere in my source, so the bug was quite confusing.

Oh God! I just realised this was where elephants (and useful tickets) have come to die. It's very difficult to track all of the tickets that are mentioned here because you have to carefully read every comment to work out if it's a comment about a ticket or a new ticket appearing. Since they're no longer open, it's even harder to track them any other way. It's almost the exact opposite of fun. :-(

Triagers: please do NOT close any more tickets as dupes of this one. If they have Unicode in the title or keywords (or codec or utf8), I'll find them and be able to work through things methodically. So better to put "unicode" in the keywords field.

No need to do anything with the tickets that are mentioned here, though, since I've got them on a list now.

Almost of the tickets mentioned here have now been fixed on the unicode branch.

The exceptions are enhancements to slug-generation (#2276, #2716), which has to be done both in Javascript and Python. #2276 is the wrong solution altogether and #2716 is a partial fix that I'll combine with some old Javascript improvements from Bill de hÓra. But that's an enhancement (it's a slug generator, not a transliteration function, after all).

I'll close this ticket once the unicode branch has been merged into trunk.

(In [5609]) Merged Unicode branch into trunk (r4952:5608). This should be fully backwards compatible for all practical purposes.

Fixed #2391, #2489, #2996, #3322, #3344, #3370, #3406, #3432, #3454, #3492, #3582, #3690, #3878, #3891, #3937, #4039, #4141, #4227, #4286, #4291, #4300, #4452, #4702