Handling gzip-compressed (or Not) Content in Python

Album Cover: Kid A

"We got heads on sticks. You got ventriloquists."
Radiohead / Kid A

Posted on June 02, 2012 10:11 PM in Programming
Warning: This blog entry was written two or more years ago. Therefore, it may contain broken links, out-dated or misleading content, or information that is just plain wrong. Please read on with caution.

I was recently working with an HTTP-based API that supported gzip compression. In trying to figure out how best to request and then handle gzip compressed data in a Python script, I ran across a Dive Into Python entry and a discussion on the subject at Stack Overflow. While the former goes into great detail about all aspects of requesting the compressed data and then handling it when it's returned and the latter is largely based on that same information, neither shed any light on what to do if the data comes back uncompressed for some reason. That's a fairly significant oversight because, as a user agent, you can request gzip compressed data as often as you like, but the web servers you request it from aren't guaranteed to return it that way.

In attempting to paint the rest of the picture, I came up with the following function for requesting gzip compressed data and handling the content that's returned whether it's compressed or not:

import gzip
from StringIO import StringIO
import urllib2

# helper function for fetching content from a URL
def fetch_url(url):

  # attempt to fetch the URL's contents with gzip compression
  request = urllib2.Request(url)
  request.add_header('Accept-encoding', 'gzip')
  response = urllib2.urlopen(request)

  # get ready for the content
  content = ''

  # if the response is gzip-encoded as expected
  if response.info().get('Content-Encoding') == 'gzip':

    # read the encoded response into a buffer
    buffer = StringIO(response.read())

    # gzip decode the response
    f = gzip.GzipFile(fileobj=buffer)

    # store the result
    content = f.read()

    # close the buffer
    buffer.close()

  # else if the response isn't gzip-encoded
  else:

    # store the result
    content = response.read()

  # return the content
  return content

I tested the function with and without the add_header() line and it worked as desired in both cases. Hopefully this will prove useful for someone else.

Comments

High PR Manual Blog comments on September 05, 2016 at 2:53 AM:

Much thanks to you such a great amount for sharing this incredible blog.Very motivating and accommodating too.Hope you keep on sharing a greater amount of your ideas.I will love to peruse.

Permalink

instagram followers on September 28, 2016 at 3:24 AM:

New site is solid. A debt of gratitude is in order for the colossal exertion.

Permalink

instagram likes on September 28, 2016 at 3:24 AM:

I read that Post and got it fine and enlightening.

Permalink

web site timesharedonations.us on October 06, 2016 at 5:58 AM:

Stunning! This could be a standout amongst the most helpful online journals we have ever run over on thesubject. Really brilliant information! I'm additionally a specialist in this subject so I can comprehend your exertion.

Permalink

Cat Spraying No More Review on October 09, 2016 at 10:31 PM:

I have read your article, it is very informative and helpful for me.I admire the valuable information you offer in your articles. Thanks for posting it..

Permalink

netflix gift on October 17, 2016 at 2:40 AM:

You are very brave focuses in this article. I would have never viewed as any of these in the event that I didn't go over this. Much obliged!. minecraft free download for pc netflix special offer code

Permalink

netflix gift on October 17, 2016 at 2:41 AM:

You are very brave focuses in this article. I would have never viewed as any of these in the event that I didn't go over this. Much obliged!. minecraft free download for pc netflix special offer code

Permalink

netflix gift on October 17, 2016 at 2:42 AM:

You are very brave focuses in this article. I would have never viewed as any of these in the event that I didn't go over this. Much obliged!. minecraft free download for pc netflix special offer code

Permalink

netflix gift on October 17, 2016 at 2:44 AM:

This article gives the light in which we can watch the truth. This is extremely pleasant one and gives indepth data. Much obliged for this pleasant article. minecraft free download for pc netflix special offer code

Permalink

voyance par telephone on December 09, 2016 at 2:42 AM:

Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can't wait to read lots of your posts.

Permalink

diabetes tipos on January 25, 2017 at 12:30 PM:

Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can't wait to read lots of your posts.

Permalink

voyance telephone on January 31, 2017 at 11:44 PM:

Great post, and great website. Thanks for the information!

Permalink

men's undershirts long on February 01, 2017 at 11:36 PM:

That is really nice to hear. thank you for the update and good luck.

Permalink

Dofollow Blog Comments on February 05, 2017 at 9:52 PM:

This is an incredible article a debt of gratitude is in order for sharing this educational data. I will visit your site consistently for some most recent post. I will visit your online journal routinely for Some most recent post.

Permalink

James Cangelosie on February 19, 2017 at 10:15 PM:

You know your projects stand out of the herd. There is something special about them. It seems to me all of them are really brilliant! High Trust Flow Backlinks

Permalink

unjhegur on May 15, 2017 at 5:23 AM:
youtube to high quality mp3 on May 21, 2017 at 2:00 AM:

We are really grateful for your blog post. You will find a lot of approaches after visiting your post. Great work.

Permalink

plastic chess sets and boards on May 26, 2017 at 11:56 PM:

You are very brave focuses in this article. I would have never viewed as any of these in the event that I didn't go over this. Much obliged!.

Permalink

FireStarter for PC on May 30, 2017 at 8:26 AM:

Great post! I am actually getting ready to across this information, is very helpful my friend. Also great blog here with all of the valuable information you have. Keep up the good work you are doing here.

Permalink

Appvn Apk Download on June 30, 2017 at 8:12 AM:

Much thanks to you such a great amount for sharing this incredible blog.Very motivating and accommodating too.Hope you keep on sharing a greater amount of your ideas.I will love to peruse.

Permalink

Georgetown Real Estate on July 19, 2017 at 10:41 AM:

I definitely enjoying every little bit of it. It is a great website and nice share. I want to thank you. Good job! You guys do a great blog, and have some great contents. Keep up the good work.

Permalink

buy brestrogen on July 22, 2017 at 10:56 PM:

I experience considerable difficulties my considerations on substance, however I truly felt I ought to here. Your article is truly awesome. I like the way you composed this data.

Permalink

over the counter diet pills on July 24, 2017 at 3:55 AM:

I experience considerable difficulties my considerations on substance, however I truly felt I ought to here. Your article is truly awesome. I like the way you composed this data.

Permalink

TechGoo on July 28, 2017 at 6:32 AM:

I really appreciate this wonderful post that you have provided for us. I assure this would be beneficial for most of the people. TechGoo

Permalink

ClickHowTo on July 28, 2017 at 6:33 AM:

i am for the first time here. I found this board and I in finding It truly helpful & it helped me out a lot. I hope to present something back and help others such as you helped me. ClickHowTo

Permalink

Dentist in Brampton,Ontario on August 07, 2017 at 7:56 AM:

Everything was wonderful. We found our new dentist.

Permalink

mobdro apk on August 17, 2017 at 1:51 AM:

Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.

Permalink

I am doing a report on this subject. Your article is full of really useful information. I will make sure to come back to check out your posts for my next report. Security camera installation

Permalink

Tony on March 05, 2018 at 2:32 AM:

Python is a great topic on which you wrote this post here. You have mentioned some interesting facts and the differences between these two things. It really increases our knowledge and I will must share it with others.
Best supplier of ground reinforcement mesh in UK

Permalink

keyword-density-checker on March 14, 2018 at 1:54 PM:

I'm happy to see the considerable subtle element here!. keyword-density-checker

Permalink

Custom embroidered patches on March 17, 2018 at 5:14 AM:

Super site! I am Loving it!! Will return once more, Im taking your food likewise, Thanks. Custom embroidered patches

Permalink

World Diabetes Day on July 21, 2018 at 1:09 AM:

They're produced by the very best degree developers who will be distinguished for your polo dress creating. You'll find polo Ron Lauren inside exclusive array which include particular classes for men, women.

Permalink

Post Comments

If you feel like commenting on the above item, use the form below. Your email address will be used for personal contact reasons only, and will not be shown on this website.

Name:

Email Address:

Website:

Comments:

Check this box if you hate spam.