Wednesday, 14 August 2013

HttpClient to download txt file having corrupt characters

HttpClient to download txt file having corrupt characters

I'm trying to extract some txt files from a server, however the file
character set is UTF-8. My code is able to download the files, however it
also produced some weird characters.
Sydney�s Desalination Plant
If I download it directly using chrome, it displays correctly as :
Sydney's Desalination Plant
Below is my current code :
public String getURL(String url) throws Exception
{
StringBuffer result=new StringBuffer();
if(StringUtils.isNotBlank(url) && url.startsWith("http"))
{
HttpClient client = new DefaultHttpClient();
client.getParams().setParameter("http.protocol.content-charset",
"UTF-8");
HttpGet request = new HttpGet(url);
// add request header
//request.addHeader("User-Agent", "");
//request.addHeader(Content-Type: text/html; charset=UTF-8)
HttpResponse response = client.execute(request);
System.out.println("Response Code : " +
response.getStatusLine().getStatusCode());
if(response.getStatusLine().getStatusCode() == 200)
{
//System.out.println(response.getEntity().getContentType().getValue());
BufferedReader rd = new BufferedReader(
new
InputStreamReader(response.getEntity().getContent(),"UTF-8"));
//result=(EntityUtils.getContentCharSet(response.getEntity()));
boolean flagIn = false;
String sCurrentLine;
while ((sCurrentLine = rd.readLine()) != null)
{
//if(flagIn==false)
//{
// sCurrentLine = removeUTF8BOM(sCurrentLine);
//}
if(flagIn)
{
result.append("\n");
}
result.append(sCurrentLine);
flagIn = true;
}
}
}
return result.toString();
}
and below is the method that tried to invoke :
System.out.println(former.getURL("http://photos.gcdis-india.com/png/bio/QSPNGC1002.txt"));
Any idea on which part I should fix? Do I need to provide any special http
header?

No comments:

Post a Comment